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Executive  Summary 


The  science  and  technology  (S&T)  program  supported  by  the  Department  of  Defense  (DOD)  is  a 
large  undertaking  (~$1  IB  in  2004)  aimed  at  ensuring  continued  technological  superiority  of  the 
U.S.  military.  Exercising  stewardship  over  this  program  is  of  great  importance.  In  this  regard  the 
program  is  subjected  to  numerous  reviews  dealing  with  its  relevance,  viability  and  productivity. 
At  various  levels  of  detail,  every  element  of  the  program  is  reviewed.  However,  the  sum  of  all 
the  reviews  does  not  constitute  an  assessment  of  the  entire  DOD  S&T  enterprise.  This  study 
examines  why  this  is  the  case  and  examines  the  prospects  for  rectifying  this  situation. 

An  effective  enterprise-wide  assessment  (EWA)  would  contribute  to  the  ability  of  the  Director  of 
Defense  Research  and  Engineering  (DDR&E)  to  provide  oversight  to  enhance  the  content  of  the 
S&T  portfolio  through  metrics  of  program  quality,  performance,  and  relevance.  It  would  increase 
collaboration  between  projects  with  aligned  objectives  and  provide  a  knowledge  base  of  the  S&T 
program  sufficient  to  justify  the  investment  to  the  services  and  Congress.  An  effective  EWA 
would  also  support  the  services  in  that  it  would  focus  their  internal  topical  reviews  to  problem 
areas  identified  in  the  EWA.  It  would  also  provide  them  a  broader  insight  into  the  entire  S&T 
program  and  their  place  within  it.  This  would  result  in  greater  program  coordination  and  reliance 
on  other  service  programs.  Also,  an  EWA  could  form  the  basis  of  an  interrelated  community  of 
experts  across  the  entire  S&T  spectrum. 

Principal  Findings 

•  Many  different  and  valuable  reviews  of  the  DOD  S&T  program  are  conducted.  However, 
there  is  no  obvious  way  (short  of  drastically  increasing  the  resources  available  to  the 
review  process)  to  modify  the  current  review  structure  so  as  to  produce  a  coherent  EWA. 
The  underlying  problem  is  the  sheer  magnitude  and  breadth  of  the  program.  This  results 
in  an  inability  to  assess  it  in  scientific  and  technical  detail  and  to  simultaneously  examine 
the  entire  program.  We  refer  to  this  conundrum  as  the  S&T  assessment  uncertainty 
principle.  A  new  methodology  must  be  developed  to  overcome  this  uncertainty  principle 
if  an  EWA  is  to  be  institutionalized. 

•  The  current  documentation  of  the  DOD  S&T  program  does  not  reflect  the  state  of  the  art 
of  information  technology,  is  marginal  for  exercising  stewardship  over  the  program,  and 
is  inadequate  for  the  conduct  of  an  EWA. 

•  DOD  does  not  access  systematically  the  breadth  of  scientific  and  technical  knowledge 
and  expertise  that  is  potentially  available  to  it.  The  establishment  of  and  ready  access  to 
communities  of  experts  that  cover  all  DOD  areas  of  scientific  and  technical  interest 
would  be  of  great  value  in  itself  and  will  be  required  for  an  EWA. 

•  The  current  state  of  the  art  and  expected  advancements  in  information  technology  and 
library  science  offer  the  best  hope  of  resolving  the  above  concerns. 
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•  An  EWA  will  be  viable  only  if  it  adds  value  to  the  parties  involved  and  emerges  as  a 
byproduct  of  the  nonnal  process  of  doing  business. 

Principal  Recommendations 

•  The  principal  investigators  of  the  various  DOD  S&T  projects/work  units  should  be  taught 
and  required  to  document  their  projects/work  units  in  tenns  of  the  very  specific  CEST 
issues  that  they  are  funded  to  resolve.  These  issues  should  be  stated  at  the  level  at  which 
people  are  actually  perfonning  work  rather  than  in  terms  of  general  objectives. 

•  S&T  project  documentation  requirements  should  be  standardized  across  DOD  and  should 
contain  the  information  needed  for  an  EWA.  Funding  for  a  project/work  unit  should  not 
be  released  until  the  organization  responsible  for  the  project/work  unit  submits 
satisfactory  documentation  to  the  Defense  Technical  Information  Center  (DTIC).  The 
DTIC  database  that  results  from  proper  documentation  of  the  program  will  be  broadly 
valuable  across  DOD  (DDR&E,  funding  agencies,  performers,  etc.)  independent  of  the 
EWA.  The  needed  information  is  available  at  the  performer  level  and  can  and  should  be 
documented  routinely.  This  documentation  is  required  as  a  matter  of  stewardship  and  will 
be  needed  for  an  EWA.  An  essential  element  of  the  documentation  is  a  succinct  statement 
of  the  critical  enabling  S&T  (CEST)  issues  being  addressed  by  the  funded  program  so 
that  an  expert  in  the  specific  topical  area  being  pursued  can  quickly  assess  the  viability  of 
the  project/work  unit  meeting  stated  plans  and  milestones. 

•  National  communities  of  experts  should  be  established  for  each  of  the  specific  CEST 
issues  being  addressed.  The  tools  of  information  technology  and  library  science  must  be 
utilized  to  accomplish  this.  Some  DOD  research  investment  may  be  required  to  resolve 
DOD  specific  issues  in  this  regard.  The  establishment  of  the  communities  of  experts  will 
be  of  great  value  in  itself  and  will  be  required  for  an  EWA. 

•  For  an  EWA,  the  approach  of  collecting  the  review  teams  at  a  central  location  for  the 
purpose  of  holding  hearings  must  be  abandoned  and  replaced  by  a  distributed  assessment 
process  that  exploits  the  documentation  of  all  of  the  CEST  issues,  advances  in 
information  technology  and  library  science,  and  the  readily  available  access  to 
geographically  distributed  communities  of  experts.  The  EWA  should  be  conducted  as  a 
targeted  survey  of  the  appropriate  communities  of  experts. 

•  A  full  EWA  as  envisioned  by  this  study  should  not  be  initiated  in  the  short  tenn.  Many 
details  need  to  be  addressed  before  such  an  assessment  could  be  undertaken.  Certain 
aspects  of  the  proposed  process  should  be  examined  via  the  conventional  review  process. 
For  example,  automated  techniques  for  forming  communities  of  experts  could  be  tested 
by  employing  them  as  part  of  establishing  the  conventional  review  teams.  Test  cases 
should  be  conducted  to  clarify  how  to  document  the  program  in  terms  of  CEST  and  to 
examine  and  resolve  problems  that  confront  the  distributed  assessment  approach.  These 
test  cases  should  address  classified  distributed  assessments  (e.g.,  utilizing  SIPRnet)  and 
unclassified  distributed  assessments.  DOD  should  be  an  active  participant  in  the  national 
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efforts  that  are  underway  to  take  advantage  of  the  ongoing  revolution  in  library  science 
(e.g.,  text  retrieval). 

If  the  processes  described  in  this  study  could  be  implemented  they  would  have  a  number  of  spin¬ 
offs.  For  example,  the  supporting  DTIC  database  could  be  interrogated  so  as  to  compare  the 
actual  program  of  record  with  the  DOD  S&T  investment  strategy.  It  would  also  establish  a  broad 
database  of  experts  covering  most  of  the  S&T  areas  of  interest  to  DOD.  It  may  be  that  the  most 
valuable  result  of  this  approach  to  an  EWA  would  be  the  establishment  and  maintenance  of  a 
detailed  database  for  the  funded  S&T  program  and  the  development  of  a  large  community  of 
experts  who  have  a  connection  with  and  interest  in  DOD  S&T.  It  is  also  worth  noting  that  while 
the  proposed  methodology  is  directed  at  assessing  the  funded  program  of  record  it  could,  if 
successfully  implemented,  provide  significant  feedback  to  the  important  matter  of  program 
formulation. 
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1.  Introduction 


The  U.S.  Department  of  Defense  (DOD)  sponsors  one  of  the  world's  largest  science  and 
technology  (S&T)  programs.  In  FY  2004  the  DOD  S&T  appropriation  was  about  $11  billion. 
This  sum,  while  less  than  three  percent  of  the  DOD  budget,  is  larger  than  the  entire  defense 
budgets  of  all  but  eleven  of  the  world's  countries  and  all  but  four  of  the  NATO  nations.  This 
large  investment  has  allowed  the  United  States  to  maintain  its  technological  advantage  over 
potential  adversaries.  It  has  undoubtedly  saved  the  lives  of  tens  of  thousands  of  U.S.  men  and 
women  in  uniform  as  well  as  prevented  countless  civilian  casualties.  Because  of  its  impact,  the 
matter  of  exercising  stewardship  over  this  large  investment  is  of  great  importance.  Stewardship 
in  this  context  has  many  dimensions.  These  include,  among  others:  program  formulation;  the 
appropriation  of  the  needed  funds;  oversight  of  expenditures;  and  the  review  and  assessment  of 
the  viability  and  productivity  of  the  funded  program.  While  all  of  these  dimensions  are  important 
and  interdependent,  this  paper  focuses  on  the  latter  dimension — namely  the  review  and 
assessment  of  the  viability  and  productivity  of  the  funded  program. 

The  very  size  of  the  program  confronts  this  aspect  of  stewardship  with  a  conundrum.  This 
conundrum  can  be  illustrated  by  examining  the  scale  of  the  effort  that  would  be  required  to 
assess  the  program  in  detail  by  conventional  means  at  the  level  where  the  work  is  actually  done. 
For  S&T  work,  scientific  and  technical  teams  typically  function  in  groups  of  three  to  five  people. 
This  corresponds  to  an  average  investment  “unit”  of  about  $  1  million.  An  $  1 1  billion  S&T 
program  will  consist  roughly  of  1,100  such  units.  This  overestimates  the  number  of  units  because 
of  funds  that  are  expended  on  hardware  purchases.  However,  the  correction  for  such  purchases 
will  not  substantially  change  the  conclusions  regarding  conducting  a  complete  detailed  review  of 
the  program.  A  proper  scientific  and  technical  review  involves  examining  the  program  at  about 
the  $1  million  level.  A  conventional  review  process  would  convene  a  team  of  5-10  experts  to 
examine  the  program  in  detail.  For  each  unit  the  review  team  would  require  perhaps  one  day  of 
preparation,  one  day  of  hearings  and  interaction  with  the  performers,  and  one  day  of  travel.  A 
single  review  team  may  review  several  related  units,  however,  a  proper  expert  review  will  add 
one  day  for  each  additional  unit.  Therefore,  1,100  S&T  units  would  require  16,000  to  32,000 
reviewer  work  days  and  the  participation  of  several  thousand  reviewers.  Compensation  for  the 
review  team’s  workdays  would  cost  between  $20  million  and  $40  million.  The  travel  and  per 
diem  costs  would  be  in  the  $3  million  to  $6  million  range.  An  estimate  of  the  reviewer  cost  for 
the  complete  review  of  the  program  is  therefore,  in  the  $25-50  million  or  range. 

The  other  major  cost  lies  in  the  preparation  for  the  reviews.  The  scientists  and  engineers  to  be 
reviewed  will  begin  preparation  several  months  prior  to  the  review.  The  experience  of  the 
authors  suggests  that  each  member  of  the  S&T  unit  will  spend  5-10  days  of  preparation  for  each 
day  of  review.  Therefore,  the  low  estimate  of  the  number  of  workdays  that  would  be  spent  in 
preparation  for  a  complete  review  of  the  S&T  program  is  about  80,000  workdays.  This 
corresponds  to  a  total  preparation  cost  of  about  $100  million.  Therefore,  the  total  cost  for  a 
complete  conventional  review  would  be  in  the  range  of  $125  million  to  $150  million.  In  relative 
terms  this  is  only  one  percent  of  the  DOD  S&T  program.  In  absolute  terms  however,  it  is  a 
considerable  amount  of  money  to  spend  on  a  review  process.  In  addition  to  the  financial  costs 
one  must  also  consider  the  impact  on  the  program  itself  of  such  a  massive  review.  There  is  also 


the  matter  of  finding  several  thousand  expert  reviewers  who  are  in  a  position  to  devote  so  much 
time  to  such  reviews.  Furthermore,  there  is  the  non  trivial  matter  of  managing  the  logistical 
aspects  of  such  a  review. 

The  practical  consequence  of  the  above  is  that  the  DOD  has  been  forced  to  make  tradeoffs 
regarding  the  types  of  reviews  and  the  program  coverage  of  reviews.  A  tradeoff  can  be 
envisioned  through  the  simple  empirical  mathematical  relationship: 

AT  AP  >  C  .  (1.1) 

Here  C  is  a  constant  related  to  the  resources  (funds,  manpower,  time,  complexity,  etc.)  that  one  is 
willing  to  apply  to  the  review  process,  AT  represents  the  uncertainty  that  an  assessment  strives  to 
achieve  in  its  understanding  of  scientific  and  technical  details,  and  AP  represents  the 
corresponding  uncertainty  that  will  result  in  the  assessment’s  ability  to  understand  the  total 
program.  We  will  refer  to  this  inequality  as  the  assessment  uncertainty  principle  (in  analogy  to 
the  uncertainty  principle  of  quantum  mechanics).  The  assessment  uncertainty  principle  says  that, 
for  a  given  value  of  C  (i.e.  allocation  of  resources  to  the  review  process)  an  assessment  must 
trade  off  the  detail  at  which  the  S&T  program  is  examined  against  the  amount  of  the  program  to 
be  assessed.  A  very  detailed  examination  of  the  S&T  (small  AT)  will  generally  result  in  a  large 
uncertainty  in  the  total  program  coverage  (i.e.  large  AP).  One  can,  of  course,  simultaneously 
achieve,  in  principle,  any  desired  value  of  AP  and  AT  by  investing  sufficient  resources  to  make  C 
small  enough.  This  is  because  the  assessment  uncertainty  principle  is  not  a  physical  law  but 
results  from  limitations  of  resources  (people,  time,  funds)  available  for  conducting  assessments. 
Such  an  undertaking,  however,  requires  a  very  substantial  investment  of  resources  (funding  and 
otherwise)  that  the  DOD  has  not  been  willing  to  make. 

The  practical  consequence  of  the  assessment  uncertainty  principle  has  been  that  the  DOD 
conducts  detailed  scientific  and  technical  assessments  on  small  portions  of  the  S&T  program  and 
conducts  management  oriented  assessments  of  the  full  program  (or  large  portions  thereof).  Both 
types  of  assessment  are  quite  valuable  and  are  done  routinely  and  professionally  by  DOD.  This 
situation  does,  however,  lead  to  concerns  regarding  what  is  actually  being  done  at  the  S&T  level 
in  this  large  and  extraordinarily  important  program.  This  concern  is  felt  especially  keenly  at  the 
highest  levels  of  DOD  where  stewardship  for  the  entire  program  resides.  In  an  attempt  to  redress 
this  situation  Dr.  John  Hopps,  then  Deputy  DDR&E  and  DUSD  (Labs),  requested  that  the 
authors  attempt  to  construct  a  methodology  that  would  permit  an  enterprise-wide  assessment 
(EWA)  of  the  entire  DOD  S&T  program.  Of  special  interest  was  the  alignment  of  the  DOD 
laboratory  system  in  terms  of  its  people,  programs  and  facilities  against  the  transformational 
goals  of  OSD,  the  Services  and  the  Agencies,  ft  was  agreed  that  a  valid  assessment  of  this 
alignment  could  only  be  done  within  the  context  of  the  overall  DOD  R&E  enterprise.  The  term 
enterprise-wide  assessment  was,  therefore,  taken  to  include  all  performers  (i.e.  industry,  DOD 
labs,  universities,  etc.),  ft  is  important  to  understand  that  a  key  objective  of  the  study  was  to 
propose  a  methodology  that  provided  coverage  of  the  entire  S&T  program  and  not  just  selected 
pieces  of  the  program.  This  has  a  profound  impact  on  the  options  that  are  available.  For  example, 
techniques  have  been  developed  to  assess  the  overall  quality  of  the  program  by  using  statistical 
sampling  methods  for  selecting  the  programs  to  be  reviewed  or  for  evaluating  the  impact  of 
particular  programs  (e.g.  see  R.N.  Kostoff,  “An  Assessment  of  the  Basic  Energy  Sciences 
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Program,”  DOE/ER  -  0123,  March  1982;  R.N.  Kostoff  “The  Handbook  of  Research  Impact 
Assessment”  DTIC  Technical  Report  ADA  296021,  1997).  Such  approaches  are  very  valuable  as 
indicators  of  overall  quality.  However,  they  do  not  satisfy  the  entire  enterprise  requirement  of 
this  study,  nor  do  they  provide  an  adequate  forcing  function  regarding  the  documentation  of  the 
entire  program. 

The  early  phases  of  the  study  brought  to  mind  Edison’s  famous  comment  when  asked  if  he  ever 
became  discouraged  working  so  long  without  results:  "Results?  Why  I  know  50,000  things  that 
won't  work".  In  this  study,  as  with  most  others,  it  was  found  that  things  are  the  way  they  are  for  a 
reason.  The  first  major  finding  of  the  study  was  that  the  database  needed  to  undertake  such  an 
assessment  did  not  exist.  It  had  been  hoped  that  the  DD1498  documentation  would  provide  a 
rudimentary  database  for  some  preliminary  work  and  form  a  base  from  which  to  evolve  the 
required  documentation.  However,  the  maintenance  of  that  database  was  found  to  be  no  longer 
required.  That  was  unfortunate,  because  that  database  was  the  only  one  that  provided  any 
detailed  scientific  and  technical  infonnation  on  the  entire  program.  It  was  quickly  realized  that 
an  entirely  new  database  would  be  required  to  support  an  EWA.  This  led  to  inquiries  into 
whether  the  data  formats  for  the  conventional  review  processes  and  the  methodology  used  by 
those  processes  could  form  the  basis  for  an  EWA.  It  ultimately  became  clear  that  the 
conventional  review  processes  could  not  be  morphed  into  a  tool  for  an  EWA.  The  S&T 
assessment  uncertainty  principle  simply  would  not  allow  it.  Fortunately,  unlike  the  uncertainty 
principle  of  quantum  mechanics,  the  S&T  assessment  uncertainty  principle  is  not  a  law  of  nature. 
It  is  an  empirical  relationship  that  is  determined  by  the  assessment  process  itself  and  the  tools 
that  are  employed  to  support  that  process.  The  study  examined  what  it  was  about  the 
conventional  assessment  methodology  that  resulted  in  inequality  (1.1).  It  became  clear  that  the 
ponderous,  manpower-intensive  character  of  the  conventional  review  process  and  the  current 
methods  of  program  articulation  and  documentation  were  among  the  major  driving  factors  that 
impose  the  inequality  and  determine  the  constant  C. 

After  considerable  deliberation  regarding  alternate  approaches,  the  study  came  to  the  conclusion 
that  the  most  promising  approach  to  an  EWA  involves  assessing  the  entire  program  at  a  fine 
level  of  scientific  and  technical  detail.  This  contradicts  the  assessment  uncertainty  principle 
(unless  one  is  willing  to  drastically  increase  the  resources  available  for  the  review  process)  and  is 
completely  orthogonal  to  the  conventional  review  processes.  This  somewhat  surprising 
conclusion  has  its  basis  in  the  reality  of  how  one  actually  decides  whether  or  not  a  particular 
scientific  and  technical  approach  to  address  a  particular  problem  has  merit.  In  the  early  phases  of 
decision  one  does  not  convene  a  massive  review.  Rather,  one  identifies  one  or  perhaps  several 
individuals  who  have  detailed  knowledge  and  specific  expertise  in  the  scientific  and  technical 
areas  being  proposed.  A  brief  conversation  (by  phone  or  meeting)  is  held  during  which  the  expert 
is  given  a  brief  description  of  the  proposed  approach  and  asked  to  provide  an  "on  the  spot" 
opinion  regarding  the  viability  of  the  proposed  approach.  If  the  individual  is  truly  expert  and  if 
the  problem/approach  is  carefully  and  succinctly  stated  then  the  "on  the  spot"  opinion  is  often 
remarkably  accurate.  The  secret  of  success  here  is  to  find  the  right  individuals  who  truly 
understand  the  specific  S&T  involved  and  to  pose  the  question  to  them  properly  and  succinctly. 

The  study  ultimately  focused  on  how  the  above  approach  might  be  extended/institutionalized  for 
the  purpose  of  an  EWA.  The  remainder  of  this  paper  offers  some  observations  and  conclusions  in 
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this  regard.  The  keys  to  success  for  an  EWA  will  be  found  in  the  proper  documentation  of  the 
program,  the  careful  and  succinct  identification  of  the  CEST  issues  that  the  projects  are  funded  to 
resolve,  the  ability  to  rapidly  establish  "communities  of  experts"  in  these  CEST  areas,  the  use  of 
a  distributed  assessment  process  employing  survey  techniques,  and  the  employment  of  advanced 
information  S&T  (most  especially  advances  in  library  science).  Advances  in  library  science  are 
now  occurring  at  a  remarkable  pace  and  will  change  forever  how  S&T  knowledge  is  accessed 
and  how  S&T  programs  are  assessed.  It  is  significant  that  each  of  these  “keys  to  success”  would 
be  valuable  to  the  DOD  S&T  program  independent  of  an  EWA.  It  is  also  worth  noting  that, 
while  the  proposed  methodology  is  directed  at  assessing  the  funded  program  of  record  it  could,  if 
successfully  implemented,  have  significant  feedback  on  the  important  matter  of  program 
formulation. 
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2.  Relationship  of  Current  DOD  S&T  Documentation  to 
Proposed  Assessment 


Proper  documentation  of  the  DOD  S&T  program  plays  an  essential  role  in  the  assessment 
methodology  proposed  in  this  paper.  This  section  briefly  examines  current  documentation  from 
the  perspective  of  what  this  study  recommends  regarding  documentation.  An  essential  element  of 
the  on-line  assessment  approach  developed  in  this  report  is  a  searchable  database  that  defines  the 
critical  descriptive  elements  and  metrics  needed  for  an  assessment.  Chief  among  these  are  the 
objectives  of  critical  enabling  S&T  (CEST).  Furthermore,  these  CESTs  must  be  developed  at  the 
work  unit  level  for  a  rapid  and  accurate  assessment  by  an  expert  reviewer.  In  terms  of  existing 
S&T  databases,  the  authors  of  this  report  have  assumed  that,  in  order  to  properly  manage  the 
S&T  program,  the  data  needed  for  an  assessment  already  exists,  at  least  at  the  performer  level. 
The  authors  have  also  concluded  that  there  has  been  no  central,  detailed,  DOD  S&T  database  at 
the  work  unit  level  since  the  requirement  for  submission  of  the  DD  1498  data  form  was 
eliminated  a  few  years  ago.  Therefore,  the  data  at  the  work  unit  level  exists  at  the  performer  level 
but  is  not  formally  provided  to  DDR&E. 

In  response  to  the  E-Government  (E-Gov)  Act  of  2002,  DDR&E  in  May  2005  launched  an  R&E 
Portal  maintained  by  DTIC.  Presently  the  portal  will  include  links  to  the  existing  databases  that 
DDR&E  maintains,  plus  the  following. 

•  New  E-Gov  database  of  current  R&E  project  summaries 

•  Research  and  Development  Descriptive  Summaries  (RDDS) 

•  Budget  Estimate  Submission  (BES) 

•  Joint  Warfare  S&T  Plans 

•  Defense  Technology  Objectives  (DTO) 

•  Basic  Research  Plan  (BRP) 

•  Defense  Technology  Area  Plan  (DTAP) 

The  portal  will  also  include  data  on  the  S&T  workforce  and  links  to  other  DOD  news  and  data. 

The  New  E-Gov  database  of  current  project  summaries  could  be  an  appropriate  vehicle  to 
establish  the  database  required  for  this  study.  At  the  present  time,  the  data  requested  for  this 
section  of  the  report  is  designed  to  facilitate  sharing  of  general  information  on  the  program  in 
order  to  establish  contacts  between  interested  parties.  The  data  required  to  assess  the  programs, 
especially  the  CESTs,  are  not  called  for.  If  DDR&E  wishes  to  implement  the  assessment 
approach  developed  in  this  report,  the  data  call  requirements  for  the  S&T  submissions  to  DTIC 
for  the  R&E  Portal  could  be  modified  to  include  the  required  level  of  project  description  needed 
to  perform  an  assessment.  The  information  required  to  implement  the  proposed  assessment 
methodology  is  displayed  in  Appendix  A  as  a  sample  Work  Unit  Summary  form.  If  DDR&E 
alternatively  wishes  to  use  the  proposed  assessment  process  for  selected  topical  reviews  rather 
than  a  global  assessment,  individual  data  forms  can  be  required. 
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A  brief  description  of  the  other  databases  linked  to  the  E-Portal  follows.  The  RDDS  and  BES  are 
both  budget  submit  documents  used  for  the  appropriation  of  funding  by  Congress.  Work 
descriptions  are  brief  and  at  a  high  level.  CESTs  are  not  discussed  at  this  level  of  description. 
The  Joint  Warfare  S&T  Plans  focus  is  on  S&T  vision,  strategy  and  planning  with  limited 
technology  description.  The  DTOs  and  the  BRP  comprise  the  most  extensive  current  database  of 
S&T.  However,  DTOs  only  address  40  percent  of  the  program  and  are  more  focused  toward 
objectives  and  milestones  than  to  CESTs.  The  DTAP  presents  DOD  objectives  and  investment 
strategy  for  ten  technology  areas  critical  to  DOD  acquisition  looking  across  Service  and  Agency 
efforts.  As  with  the  other  documents  it  is  not  written  at  the  work  unit  level. 

There  are  a  few  other  databases  and  program  descriptions  that  are  maintained  by  DOD  or  the 
Services.  For  example,  the  DOD  maintains  a  Militarily  Critical  Technologies  List  (MCTL)  as 
part  of  the  Militarily  Critical  Technologies  Program  (MCTP).  While  the  scope  of  this  program  is 
to  determine  existing  technologies  of  use  to  potential  adversaries  there  are  features  of  this 
program  of  interest  to  this  report.  In  particular  the  MCTP  maintains  a  roster  of  about  1,000 
subject  matter  experts  from  the  military  Services,  DOD,  other  federal  agencies,  industry,  and 
academia.  These  subject  matter  experts  are  analogous  to  the  community  of  experts  proposed  to 
assess  the  DOD  S&T  enterprise.  The  MCTP  however  does  not  use  search  engine  technology  to 
find  experts  and  the  scope  of  the  MCTL  is  only  a  subset  of  the  DOD  S&T  enterprise.  Another 
feature  of  the  MCTL  is  the  generation  of  data  sheets  that  require  Critical  Technology  Parameters 
that  are  defined  at  the  same  level  as  this  report  recommends.  Due  to  the  nature  of  the  MCTL 
however,  these  Critical  Technology  Parameters  represent  existing  technology  and  are  not  future 
project  objectives  as  defined  in  this  study. 

Each  of  the  Services  documents  its  own  programs.  There  is  no  common  format  and  this 
information  is  not  generally  distributed.  The  Anny  produces  a  fairly  comprehensive  two  volume 
report  on  its  S&T  program  entitled  Army  Science  &  Technology  Master  Plan.  While  this  plan  is 
comprehensive,  it  only  gives  project  descriptions  for  the  Army  Technology  Objectives  (ATOs 
were  adopted  by  DDR&E  as  DTOs).  As  with  DTOs,  the  AROs  only  represent  part  of  the  Army 
program  and  do  not  provide  the  CESTs  necessary  for  the  proposed  assessment  methodology. 

NASA  has  generated  a  promising  database  that  has  provided  a  rough  template  for  this  study.  A 
brief  description  of  the  NASA  Technology  Inventory  will  be  found  at  Appendix  C. 

A  key  element  for  an  EWA  is  a  searchable  database  of  the  entire  program  that  is  generated  at  the 
performer  level.  A  comprehensive,  searchable  database  that  documents  the  DOD  S&T  program 
at  the  performer  level  should  exist  independent  of  the  desire  for  an  EWA.  Technology  allows  for 
the  creation  of  such  a  database;  routine  documentation  of  the  program  at  the  performer  level 
pennits  it  and  stewardship  for  the  program  requires  it. 

This  study  concludes  that  a  comprehensive  programmatic  database,  similar  to  the  fonn  DD1498, 
be  established  as  a  requirement  for  work  units  in  DOD  and  be  maintained  by  DTIC.  Aside  from 
the  necessity  of  a  comprehensive  database  to  perfonn  a  meaningful  EWA,  better  program 
documentation  will  provide  other  benefits  such  as  more  effective  development  of  cohesive  major 
S&T  initiatives  such  as  development  of  hypersonic  flight  vehicles. 


6 


The  authors  contend  that  the  infonnation  to  generate  an  assessable  database  exists  at  the 
performer  level,  and  the  electronic  means  to  collect  and  distribute  the  information  needed  for  an 
assessment  exists  at  DTIC  through  the  E-Portal  for  S&T.  However,  the  requirement  to  provide 
DDR&E  with  this  level  of  work  unit  detail  does  not  exist.  Such  a  requirement  is  a  necessary 
component  of  an  EWA. 
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3.  Established  Review  Processes 


Background 

The  purpose  of  this  section  is  to  briefly  describe  the  current  methods  that  the  services,  DDR&E, 
and  Congress  employ  to  assess  the  S&T  program.  As  will  be  shown,  there  are  several  review 
processes  in  place  in  the  services,  DDR&E,  and  congressional  staff.  It  is  probably  fair  to  say  that 
at  some  level  of  detail  every  element  of  the  S&T  program  funded  by  DOD  is  subjected  to  some 
type  of  review.  However,  there  is  no  uniformity  across  or  even  within  services  as  to  the  level  of 
granularity  and  the  composition  of  a  review  body.  The  purposes  of  these  individual  reviews  vary 
according  to  the  needs  and  management  structure  of  the  reviewing  organizations.  Also  there  is 
no  uniformity  in  the  documentation  of  reviews,  and  what  documentation  there  is  is  often  not 
generally  distributed  to  the  broad  S&T  community.  Therefore,  the  sum  of  all  current  reviews  and 
documentation  does  not  represent  an  enterprise  wide  assessment  (EWA). 

An  effective  EWA  would  contribute  to  DDR&E ’s  ability  to  provide  oversight  to  enhance  the 
content  of  the  S&T  portfolio  through  metrics  of  program  quality,  perfonnance  and  relevance.  It 
would  increase  collaboration  between  projects  with  aligned  objectives  and  provide  a  knowledge 
base  of  the  S&T  program  sufficient  to  justify  the  investment  to  the  services  and  Congress.  An 
effective  EWA  would  also  support  the  services  in  that  it  would  focus  their  internal  topical 
reviews  to  problem  areas  identified  in  the  EWA.  It  would  also  provide  a  broader  insight  into  the 
entire  S&T  program  and  the  place  of  the  services  within  it.  This  would  result  in  greater  program 
coordination  and  reliance  on  other  service  programs.  Also,  an  EWA  could  form  the  basis  of  an 
interrelated  community  of  experts  across  the  entire  S&T  spectrum. 

The  starting  point  for  an  EWA  is  a  review  of  the  current  review  processes  and  an  assessment  of 
strengths  and  shortcomings.  In  support  of  this  objective,  a  framework  is  presented  that 
characterizes  current  assessments  and  reviews  of  the  S&T  program  by  the  purpose  of  the  review; 
the  level  of  detail  or  granularity  of  the  subject  matter;  and  the  composition  of  the  reviewing 
body.  In  a  broad  sense,  program  reviews  can  be  divided  into  two  categories:  planning  reviews, 
where  the  objective  is  to  assess  a  program  prior  to  initiation  to  detennine  if  the  work  is  worth 
funding  (is  this  the  right  job?),  and  technical  reviews,  where  the  objective  is  to  detennine  the 
quality  and  progress  of  the  work  (is  the  job  being  done  right?).  A  further  objective  is  to 
determine  if  the  work  is  legitimately  S&T  or  should  be  a  higher  category  of  funding.  The 
program  assessment  methodology  proposed  in  this  study  is  focused  on  the  technical  reviews  and 
assumes  that  the  objectives  of  the  program  are  in  line  with  the  needs  of  the  military  users.  A  third 
type  of  review  that  evaluates  completed  work  done  in  the  past  provides  hindsight  lessons  learned 
and  program  accomplishments.  This  type  of  review  is  not  germane  to  this  report. 

The  following  sections  will  include  a  broad  description  of  the  established  review  processes, 
followed  by  the  details  of  current  reviews  by  the  services,  DDR&E,  and  congressional  staff.  This 
will  be  followed  by  commentary  on  strengths  and  shortcomings  and,  finally,  by  conclusions. 
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Types  of  Reviews  and  Purpose 


Table  3.1  portrays  the  current  types  of  reviews  in  effect  for  the  DOD  S&T  program.  The  three 
columns  are  in  decreasing  order  of  granularity  and  increasing  order  of  comprehensiveness.  The 
first  column,  Technical  Subject,  examines  a  project  at  the  level  where  the  work  is  actually  done. 
The  responsible  briefer  to  a  review  panel  is  generally  the  principal  investigator.  Because  of  the 
many  thousands  of  S&T  projects  at  this  level  of  granularity,  the  entire  program  cannot  be 
reviewed  at  this  level  using  conventional  review  procedures.  At  best,  20  percent  of  the  S&T 
program  is  currently  evaluated  at  this  level.  Increasing  this  fraction  significantly  would  place 
unacceptable  demands  in  tenns  of  time  and  money  on  performers  and  reviewers.  In  addition, 
there  are  difficulties  in  assembling  a  review  board  with  enough  members  that  there  is  sufficient 
member  expertise  as  well  as  a  balance  of  strong  opinions  predicated  on  the  members’  work  or 
bias  in  the  subject  being  reviewed.  Therefore,  the  current  Technical  Subject  Reviews,  while 
sufficient  in  detail,  lack  the  comprehensiveness  for  an  EWA  and  are  difficult  and  time 
consuming  to  conduct.  This  fact  leads  to  consideration  of  a  new  type  of  review  process,  as 
discussed  subsequently  in  this  report,  that  aims  to  provide  both  the  detail  and  comprehensiveness 
that  lead  to  an  EWA. 


Table  3.1  Types  of  Conventional  Reviews  and  Purpose 


Types  of  Reviews 


Technical 

Subject 

Subject 

Area 

Overview 

Investment 

Oversight/ 

Audit 

Service  Internal 
Management 

(peer  level  technical 
experts  & 
management) 

Technical  quality 

Technical 

progress 

Resource 

balance 

between 

projects 

Program  execution, 
obligation,  and 
expenditure 

Technology  User 
Community 

(customers  of 
technology 
e.g.  Syscoms  & 
operational  forces) 

Response  to 
requirements 
and  emerging 
transition 
opportunities 

Department  of 
Defense 

(DDR&E) 

Ad  hoc  issue 
resolution 

Program 

oversight 

(TARA) 

Program  execution, 
obligation,  and 
expenditure 

Congress 

(Congressional 
Armed  Services  staff) 

Ad  hoc,  special 
interest,  topical 

Program 
element 
funding  level 
adjustment 

Program  element 
funding  level 
adjustment 
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Despite  the  difficulties  inherent  in  conducting  an  EWA  at  the  scientific  and  technical  subject, 
project,  or  work  unit  level  (the  terms  are  used  interchangeably  here),  this  is  the  appropriate  level 
at  which  to  evaluate  the  program.  In  S&T,  the  devil  is  in  the  details.  The  work  unit  is  where  the 
work  actually  gets  done.  At  this  level,  an  expert  in  the  technical  discipline  under  review  can 
readily  assess  the  technical  merit  of  the  approach  and  the  feasibility  of  meeting  time  and  cost 
metrics,  evaluate  whether  the  work  is  S&T  in  character,  and  evaluate  the  competence  of  the 
performers.  Reviews  held  at  higher  levels  of  aggregation  serve  other  purposes,  such  as  ensuring 
program  balance  and  response  to  requirements.  Reviewers  of  programs  at  higher  levels  of 
aggregation  are  also  often  reviewing  projects  in  which  they  are  not  experts.  This  leads  to  longer 
discussion  and,  therefore,  more  time  needed  to  reach  decisions.  Only  the  subject  level  review 
gets  at  the  actual  details  of  the  work  with  specialist  reviewers,  who  can  usually  quickly  and 
accurately  determine  the  value  of  the  work.  Even  at  this  level,  the  small  number  of  specialists 
can  result  in  parochial  interests  influencing  the  review. 

The  next  column,  Subject  Area  Overviews,  represents  the  collection  of  S&T  projects  that  he  in  a 
common  discipline.  For  example,  the  DDR&E  taxonomy  of  Platforms,  Weapons,  Space, 
Information  Technology,  Materials,  Human  Systems,  Environments,  and  Sensors/EW  forms  a 
fairly  comprehensive  technology  review  format,  although  even  at  this  level  DDR&E  does  not 
conduct  a  comprehensive  review.  Another  example  of  a  Subject  Area  Overview  is  a  review  of  a 
focused  initiative  (e.g.,  the  National  Aerospace  Initiative),  that  incorporates  many  diverse 
technologies.  At  this  level  of  granularity,  much  of  the  S&T  program  can  be  reviewed,  although 
usually  not  in  a  given  year.  The  DDR&E  Technology  Review  and  Assessment  (TARA)  process 
has  reviewed  the  program  at  the  Subject  Area  level  on  a  rolling  schedule  of  half  the  program 
every  year  (this  process  is  under  review  and  is  subject  to  change).  These  reviews  are  a  tradeoff 
between  comprehensiveness  and  level  of  detail  and  represent  DDR&E ’s  current  review 
management  tool.  However,  for  an  EWA,  the  TARA  process  fails  to  provide  both  enough  detail 
and  enough  comprehensiveness. 

The  final  column,  Investment  Oversight/Audit,  has  the  least  technical  content,  is  the  most 
comprehensive,  and  is  budgetary  in  nature.  These  reviews  mostly  deal  with  program  execution, 
funding  obligation  rates,  and  expenditures.  As  such,  they  do  not  address  the  level  of  program 
detail  necessary  for  an  EWA. 

The  four  rows  represent  an  ascending  managerial  level  of  program  oversight.  Program 
development  and  execution  responsibility  is  primarily  at  the  service  level,  so  the  most  detailed 
and  comprehensive  reviews  are  conducted  at  this  level.  The  services  review  the  program  at  many 
levels  of  granularity  including  technical  subject  reviews  focused  on  the  quality  of  the  work,  the 
relevance  of  the  work  to  service  needs,  the  progress  of  the  work  in  terms  of  cost  and  milestones, 
and  evaluate  whether  the  work  is  really  S&T  or  in  fact  a  higher  category  of  funding.  Technical 
experts  and  stakeholders  in  the  program  generally  conduct  these  reviews.  As  mentioned  earlier, 
there  is  no  consistency  in  these  reviews  across  services  and  even  within  services.  The  conduct  of 
these  reviews  is  generally  given  to  the  first  or  second  level  of  service  program  management.  Due 
to  the  overall  size  of  even  the  individual  service  programs,  senior  service  S&T  managers  are 
usually  involved  only  in  selected  topical  reviews  and  in  issue  resolution  at  the  technical  subject 
level.  Senior  service  S&T  oversight  is  more  generally  focused  at  the  Subject  Area  level  with  the 
objective  of  decisionmaking  across  S&T  priorities.  The  technology  user  communities,  the 
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customers  of  technology,  are  usually  more  involved  in  program  formulation  and  approval  than  in 
technical  quality  and  execution  reviews.  The  user  community’s  primary  concern  is  that 
technology  is  addressing  service  requirements.  In  addition,  however,  users  are  involved  in 
program  reviews  of  technologies  near  completion  and  transition  to  engineering  development. 
These  are  generally  6.3  advanced  development  reviews. 

The  preceding  discussion  of  the  customers  and  audience  for  current  reviews  raises  the  associated 
issue  that,  while  there  is  currently  no  EWA,  there  is  also  no  current  service  requirement  or 
ownership  for  this  product.  To  be  useful,  an  EWA  would  need  to  strengthen  the  service  programs 
but  not  significantly  increase  time  and  money  spent  on  reviews.  In  this  regard,  the  EWA 
proposed  here  might  actually  reduce  the  number  of  conventional  topical  reviews  by  highlighting 
only  the  potential  problem  areas  for  topical  reviews. 

Conventional  Review  Process 

While  table  3.1  portrays  a  matrix  of  types  of  reviews  and  purposes  of  reviews  as  well  as 
differences  in  the  details  of  individual  review  processes,  there  are  common  elements  that 
conventional  reviews  share.  First,  an  external  review  needs  to  be  distinguished  from  routine 
supervisory  management.  The  principal  distinguishing  factor  of  the  external  review  is  the  review 
board  composed  of  individuals  outside  the  project  being  reviewed  and  selected  from  the 
community  of  experts  in  the  discipline  to  be  reviewed.  The  purpose  of  the  review  board  is  to 
strive  for  impartial  judgment  as  well  as  an  external  perspective.  For  program  credibility  and 
program  defense,  an  external  review  is  essential.  For  a  completely  independent  review,  the 
review  can  be  “contracted  out”  to  an  organization  such  as  the  National  Research  Council  (NRC). 
Each  of  the  services  maintains  study  boards  (e.g.,  the  Anny  Study  Board)  that  generally  do 
studies  and  analyses  but  also  provide  program  review  boards.  Predominately,  however,  the 
services  develop  their  own  stable  of  external  reviewers.  These  reviewers  are  often  technical 
experts  with  previous  professional  experience  within  the  services  programs.  Retired  military 
officers  are  also  often  solicited  for  their  background  in  service  requirements. 

The  selection  of  the  actual  reviewers  is  critical  to  any  review  process  and  is  a  core  issue  of  the 
EWA  process  analyzed  in  this  report  in  which  potential  reviewers  are  identified  by  a  computer 
search  based  on  CEST  objectives  associated  with  the  project  to  be  reviewed.  Individuals  with 
technical  competence  developed  through  work  in  any  field  will  have  biases  shaped  by  their 
experience  and  affiliations.  To  compensate  for  this,  a  sufficient  pool  of  reviewers  is  required  to 
balance  out  bias  and  provide  a  level  of  statistical  sampling.  Many  current  reviews  suffer  from 
choosing  reviewers  too  close  to  the  program  that  do  not  subject  the  program  to  difficult 
questions. 

External  reviews  tend  to  be  very  time  consuming,  hence  the  difficulty  of  performing  an 
enterprise  assessment.  For  an  important  review,  one  that  can  influence  the  direction  and  the 
funding  of  the  project  being  reviewed,  significantly  more  time  is  spent  in  preparation  of  briefing 
material,  analysis,  and  documentation  of  results  than  in  the  review  itself.  Frequently,  reviews 
require  presentations  in  a  prescribed  format,  resulting  in  the  creation  of  dedicated  briefing 
materials.  The  presenters  often  give  dry  runs  of  their  presentations  to  their  management  and  may 
redo  their  presentations  as  a  result.  Reviewers  are  often  given  read-ahead  material  to  prepare 
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them  for  the  briefings.  Scoring  sheets  are  usually  developed  to  correlate  reviewer  comments. 
Following  a  review,  the  review  board  customarily  provides  preliminary  feedback  followed  by  a 
report. 

The  DOD  S&T  enterprise  now  exceeds  $1  IB  a  year.  If  a  representative  project  or  work  unit  were 
about  $1M  a  year  (three  or  four  investigators,  plus  project  expenses)  11,000  projects  would 
require  reviews.  DOD  has  not  considered  it  a  priority  to  perfonn  11,000  reviews  at  the  project 
level  and,  therefore,  from  a  corporate  viewpoint,  only  looks  at  a  fraction  of  its  projects  (about  20 
percent)  at  a  detailed  level  each  year.  The  enterprise  is  only  viewed  in  its  entirety  in  very  large 
chucks  at  the  program  element  level.  An  EWA  at  the  detailed  project  level  will  require  a  process 
different  from  the  conventional  external  review. 

Service  and  DDR&E  Reviews 

Each  of  the  services  manages  its  S&T  programs  differently,  particularly  the  6.2  Applied 
Research  program  and  the  6.3  Advanced  Development  program,  but  each  service  has  a  research 
office.  The  Army  Research  Office  (a  subordinate  organization  of  the  Army  Research  Lab),  is 
under  the  authority  of  the  Deputy  Assistant  Secretary  for  Research  and  Technology  who  reports 
to  the  Army  Acquisition  Executive.  The  director  of  the  Air  Force  Office  of  Science  Research 
reports  to  the  Air  Force  Research  Laboratory  Commander.  The  Chief  of  Naval  Research  reports 
to  the  Assistant  Secretary  of  the  Navy  for  Research,  Development  and  Acquisition.  The  Navy 
has  had  a  tradition  of  funding  a  larger  6.1  account  than  the  other  services,  and  it  manages  all  of 
its  S&T,  6. 1-6.3,  under  the  Chief  of  Naval  Research,  Office  of  Naval  Research.  The  Marine 
Corps  S&T  program,  a  small  component  of  Naval  Research  (~  $30M  /  year),  is  also  managed  at 
the  Office  of  Naval  Research  to  provide  synergy  with  related  Navy  S&T.  The  Air  Force  S&T 
program  is  executed  by  the  Air  Force  Research  Laboratory.  The  Army  does  not  centrally  manage 
S&T  programs  under  a  single  organization.  The  Army  has  four  Major  Commands  and  one  Field 
Operating  Agency  of  Headquarters  DA  that  own  all  of  the  laboratories.  The  predominant  funding 
for  Army  S&T  is  managed  by  Army  Materiel  Command’s  Research  Development  and 
Engineering  Command,  approximately  80  percent  of  the  $  1 .7B  annual  Anny  S&T  budget.  Of  the 
three  services,  the  Air  Force  has  given  the  highest  priority  to  science  and  engineering  education 
to  its  military  officers  hence  the  management  and  execution  of  the  Air  Force  programs  utilizes  a 
higher  proportion  of  service  men  and  women  than  the  other  services  (with  the  Navy  having  the 
highest  proportion  of  civilians).  With  different  management  structures  and  personnel  among  the 
services  it  is  to  be  expected  that  different  assessment  processes  are  in  place  together  with 
different  documentation  procedures. 

Army  Review  Process.  Most  Anny  6.2  and  6.3  programs  are  managed  as  Army  Technology 
Objectives  (ATOs)  (DDR&E  has  adopted  the  Army  framework  of  ATOs  as  DTOs  for  its  own 
TARA  reviews).  These  ATOs  are  3-5  years  in  length  and  have  defined  milestones  and 
deliverables.  The  Army  reviews  these  programs  annually  at  the  Anny  S&T  Executive  level  in  a 
largely  internal  review  process.  The  remainder  of  the  Army  S&T  program  is  managed  at  the 
Army  Research  and  Development  Center  and  Laboratory  level,  generally  employing  a  peer 
review  process.  The  Anny  documents  its  program  in  the  Army  S&T  Master  Plan,  which  is 
published  biennially. 
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Air  Force  Review  Process.  The  Air  Force  has  the  most  comprehensive  review  process  for  its 
S&T  programs.  The  Air  Force  Science  Advisory  Board  reviews  the  program  assessing  five  of  the 
ten  Directorates  each  year.  The  reviews  are  intensive,  generally  a  week  long,  and  the  review  of 
the  five  Directorates  takes  about  two  months  every  year.  The  Basic  Research  program  is 
reviewed  by  the  Science  Advisory  Board  within  this  process  when  it  reviews  the  AFOSR  which 
is  a  directorate  in  AFRL. 

Navy  and  Marine  Corps  Review  Process.  The  Navy  differs  significantly  from  the  Army  and  Air 
Force  both  in  management  structure  of  S&T  and  consequently  in  reviews.  Also,  in  recent  years, 
the  introduction  of  the  Future  Naval  Capabilities  (FNC)  initiatives  has  changed  both 
management  and  review  processes  for  the  Navy  and  Marine  Corps.  S&T  is  centrally  managed  at 
the  Office  of  Naval  Research  for  the  Navy  and  Marine  Corps.  The  6.1  and  part  of  the  6.2 
program  are  managed  by  six  S&T  departments,  each  employing  peer  review  processes  differing 
in  detail  and  documentation.  The  FNC’s  comprise  the  6.3  program  and  the  remainder  of  the  6.2 
program  and  are  matrix  managed  in  ONR.  The  FNC’s  are  strongly  influenced  by  the  user 
community  who  are  active  in  its  review  process.  The  NRL  has  its  own  peer  review  process  for  its 
corporate  S&T  programs. 

DDR&E  Review  Process.  The  Technology  Area  Review  and  Assessment  (TARA)  process  is 
currently  DDR&E’s  principal  review  process.  TARA  review  panels  are  selected  by  DDR&E 
from  a  community  of  experts  selected  by  the  DDR&E  staff  responsible  for  a  particular  area 
review.  Because  of  the  broad  nature  of  these  reviews  the  board  tends  to  be  selected  from 
technology  generalists  familiar  with  the  DOD  S&T  program  together  with  some  specialists. 
While  these  area  reviews  are  a  week  in  length  they  only  look  in  detail  at  Defense  Technology 
Objectives  (DTOs)  that  represent  less  than  40  percent  of  the  entire  program.  Programs  are 
selected  as  DTOs  based  on  joint  service  participation  and  are  most  often  6.3  or  late  6.2  in  nature. 
While  technology  objectives  are  discussed  in  the  TARA  the  level  of  detail  is  rarely  at  the  CEST 
level. 

The  TARA  board  is  also  provided  a  broad  but  brief  overview  of  the  remainder  of  the  program  at 
these  system  level  reviews. 

Congressional  Staff  Reviews.  As  a  part  of  the  budgetary  process,  congressional  staff  members 
from  the  House  and  Senate  Armed  Service  Authorization  and  Appropriations  Committees  review 
the  service  programs  at  the  budgetary  Program  Element  level.  While  these  reviews  are 
comprehensive  they  do  not  represent  an  EWA.  Since  the  nature  of  these  reviews  is  for  funding 
justification,  the  services  use  these  reviews  to  sell  their  programs,  and  problem  areas  are 
underplayed.  Congress  is  left  to  make  funding  decisions  without  the  context  of  the  overall  needs, 
opportunities,  and  priorities  of  the  S&T  enterprise.  An  EWA  could  contribute  to  better 
congressional  understanding  of  the  S&T  program  and,  hopefully,  to  better  funding  decisions. 

Current  Review  Process  Strengths  and  Shortcomings 

With  the  exception  of  the  TARA  review,  the  current  multiple  review  processes  are  designed  to 
meet  individual  service  needs  and  management  objectives.  The  principal  strengths  of  the  current 
reviews  in  practice  are  that  the  services  take  ownership  of  them,  do  not  feel  threatened  by 
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external  oversight  (hence,  are  more  open  in  providing  information),  and  are  more  inclined  to  act 
on  internally  generated  recommendations  and  criticisms  than  on  those  recommended  by 
DDR&E.  The  service  reviews  at  the  Anny  and  Air  Force  Directorate  or  ONR  levels  are 
generally  intensive  and  employ  peer  reviewers.  An  assumption  inherent  in  this  report  is  that,  at 
the  service  performer  level,  the  perception  is  that  there  is  an  adequate  review  process  for  their 
needs,  and  additional  reviews  would  be  a  burden  with  little  benefit  to  them. 

Despite  these  strengths,  there  are  still  serious  shortcomings  in  the  current  review  process.  A 
significant  shortcoming  is  that  there  is  no  EWA  of  the  DOD  S&T  program.  The  DDR&E  TARA 
process  is  the  closest  attempt  but  is  incapable  of  examining  the  enterprise  at  a  granularity 
sufficient  to  understand  the  program  at  the  work  unit  level,  nor  is  it  comprehensive.  Since  the 
TARA  process  uses  the  conventional  review  method  of  a  review  board  convened  for  several  days 
of  review,  the  overall  size  of  the  S&T  program  is  too  large  to  address  conventionally,  even 
looking  at  only  half  of  the  program  every  year.  The  lack  of  an  EWA  means  that  overall 
judgments  of  the  program  become  qualitative,  difficult  to  put  into  perspective,  and  subject  to 
criticism.  For  example,  there  is  limited  ability  to  ensure  jointness  between  related  efforts,  and  it 
is  difficult  to  compare  work  done  by  DOD  to  that  of  other  agencies,  such  as  DOE  and  NASA  ,or 
to  other  perfonners  both  national  and  international. 

Another  major  shortfall  is  that  the  portfolio  of  current  reviews  is  conducted  at  many  different 
levels  of  granularity  with  nothing  approaching  a  common  format,  even  within  individual 
services.  Because  of  this,  the  individual  reviews  cannot  be  coherently  combined  into  a 
comprehensive  picture  of  the  enterprise.  Another  shortfall  is  that  the  documentation  of  service 
reviews  is  not  generally  available  across  DOD,  further  limiting  an  enterprise-wide  picture. 
Finally,  current  reviews  have  made  little  use  of  information  technology  tools  that  could  reduce 
travel,  speed  the  process,  correlate  material,  and  review  results. 


Conclusions  Regarding  Conventional  Reviews 

The  desire  to  introduce  an  overall  S&T  assessment  of  the  S&T  enterprise  at  a  level  of  granularity 
sufficient  to  justify  and  manage  the  program  raises  a  conundrum,  the  so-called  S&T  uncertainty 
principle.  On  the  one  hand  is  the  expectation  that  an  enterprise-wide  assessment  would 
strengthen  the  program,  explain  the  quality  and  relevance  of  the  work,  and  support  justification 
of  the  large  DOD  investment  in  S&T.  On  the  other  hand,  the  work  load  of  such  an  assessment 
using  conventional  review  techniques  would  create  a  massive  overload  of  work  on  the  performer 
base.  It  would  be  strongly  resisted  and  would  likely  prove  to  be  untenable.  The  search  for  a 
resolution  of  this  conundrum  forms  the  basis  of  this  study. 

While  extensive  review  processes  exist,  the  sum  of  these  reviews  does  not  constitute  an  EWA. 
Differences  in  granularity,  comprehensiveness,  review-board  composition,  fonnat, 
documentation  and  even  the  review  objectives  of  current  reviews  do  not  result  in  a  homogeneous 
understanding  of  the  program.  In  addition,  critical  review  material  is  generally  not  released  by 
the  services.  An  EWA  of  the  S&T  base  would  be  of  significant  value  to  the  DOD.  A  searchable 
database  coupled  with  the  powerful  search  engines  already  in  use  and  under  further  refinement 
might  offer  a  powerful  new  tool  to  DOD  for  program  assessment  while  adding  little  to  the  work 
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load  of  the  S&T  workforce.  For  example,  an  EWA  applied  to  the  major  challenge  posed  by 
Improvised  Explosive  Devices  (IED’s)  might  not  only  help  fund  promising  ideas  but  might  also 
eliminate  investment  in  technological  projects  that  experts  would  rule  incapable  of  addressing 
the  issues.  A  key  tenet  of  this  study  is  that  the  coupling  of  the  best  review  expertise,  obtained  by 
computer  search,  to  a  work  project  completely  within  their  field  of  knowledge  will  result  in  a 
very  rapid  and  accurate  assessment  of  the  work  effort. 

An  EWA  would  create  a  vehicle  for  DOD  to  gain  access  to  a  “community”  of  scientific  and 
technical  experts  that  will  not  only  contribute  to  assessing  the  program  but  will  open  DOD  to  the 
broader  S&T  done  in  other  agencies,  the  private  sector,  and  the  global  community.  Even  if  not 
applied  across  the  entire  enterprise  (especially  in  its  first  applications),  this  computer  based 
review  methodology  might  be  of  significant  value  in  selecting  reviewers  and  accessing  related 
work  for  a  more  conventional  board  review.  In  fact,  this  application  would  be  a  good  first  test  of 
its  utility. 
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4.  An  Approach  to  Enterprise-wide  Assessment 


Background 

The  previous  section  suggests  that  it  is  unrealistic  to  approach  an  EWA  of  the  total  DOD  S&T 
Program  using  the  conventional  review  process.  The  scope  of  such  an  undertaking  would  be 
unmanageable  and,  if  done,  the  volume  of  the  resulting  reports  would  be  so  large  as  to  have  little 
management  value.  A  non-conventional  approach  must,  therefore,  be  considered.  Any 
assessment  of  the  program  must  have  available  to  it  a  database  that  documents  the  program  at  a 
level  of  detail,  a  level  of  completeness  and  in  a  format  that  is  compatible  with  the  desired 
assessment.  Therefore,  establishing  the  required  documentation  of  the  program  will  be  necessary 
to  any  EWA.  An  assessment  will  also  involve  access  to  individuals  who  are  experts  in  the  topics 
being  assessed.  A  means  of  identifying  these  experts  must  be  in  place  before  an  assessment  can 
proceed.  The  assessment  process  itself  must  be  defined  and  put  in  place.  Finally  a  means  of 
displaying  the  data  generated  by  the  assessment  must  be  developed  so  that  understanding  and 
actions  can  follow  from  the  assessment.  The  following  paragraphs  will  address  each  of  these 
requirements. 

Program  Documentation 

To  come  to  grips  with  what  documentation  is  necessary  for  an  EWA,  it  is  helpful  to  consider 
how  S&T  is  actually  done.  In  this  regard,  there  often  is  confusion  between  S&T  and  capability. 
This  results  in  a  tendency  to  document  the  program  in  terms  of  desired  capability  rather  than  in 
tenns  of  the  details  of  the  S&T  that  is  actually  being  conducted.  As  an  illustration  of  this  point, 
consider  that  hypersonic  flight  is  often  viewed  as  a  technology.  However,  there  are  many 
different  sciences  and  technologies  that  contribute  to  hypersonic  flight.  Hypersonic  flight  is  a 
capability  rather  than  a  technology.  In  general,  scientists  and  engineers  each  work  on  a  small  part 
of  the  S&T  needed  to  achieve  the  capability  of  hypersonic  flight.  The  array  of  different  scientific 
and  technical  disciplines  needed  to  support  a  capability  like  hypersonic  flight  is  quite  large.  The 
viability  of  achieving  a  desired  capability  will  depend  on  the  viability  of  the  S&T  programs 
aimed  at  resolving  the  CEST  issues  that  prevent  one  from  just  getting  on  with  building  the 
desired  capability. 

It  should  be  clear  from  the  above  that  properly  defining,  identifying,  and  documenting  the  CEST 
issues  must  be  of  high  priority  in  preparing  for  an  EWA.  It  is,  therefore,  worth  spending  some 
time  addressing  the  matter  of  how  CESTs  should  be  defined  and  documented.  For  this  purpose, 
we  will  continue  with  the  example  of  hypersonic  flight.  We  have  chosen  this  example  because  it 
was  of  special  interest  to  the  DDR&E  at  the  time  the  study  was  commissioned,  it  was  recently 
examined  by  the  NRC  [“Evaluation  of  the  National  Aerospace  Initiative,”  National  Academies 
Press,  2004]  and  it  illustrates  the  complexity  of  properly  defining  CESTs  such  that  they  are 
useful  in  the  EWA  context.  There  are  various  levels  at  which  one  could  identify  the  CESTs 
associated  with  hypersonic  flight.  For  example,  as  mentioned  above,  one  could  define  the  CEST 
as  “hypersonic  technology.”  If  one  chooses  this  definition,  then  it  is  straightforward  to  employ 
various  readily  available  S&T  databases  and  search  engines  to  gain  information  (see  figure  4.1 
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and  tables  4.1  through  4.3).  The  searches  were  done  using  the  Engineering  Compendex  (Ei 
Compendex)  database  (http://www.ei.org/compendex.html).  Articles  related  to  hypersonic 
technology  were  identified  from  the  Ei  Compendex  database  for  the  years  1991  to  2004.  The  Ei 
Compendex  provides  abstracted  information  from  significant  engineering  and  technical  literature 
from  around  the  world.  In  addition  to  4,500  journals,  government  reports,  and  books,  the 
database  holds  over  480,000  records  from  the  published  proceedings  of  selected  conferences.  A 
more  complete  search  could  be  done,  e.g.,  by  adding  a  Science  Citation  Index 
(http://scientific.thomson.com/products/sci/)  search.  However,  for  the  purpose  of  this  discussion 
the  Ei  Compendex  search  is  adequate. 


From  the  unfiltered  query  results,  conference  papers  were  separated  from  the  total  number  of 
articles.  Several  ranking  functions  were  used  to  identify  the  most  prolific  authors,  journals, 
institutions,  and  countries  as  measured  by  number  of  journal  articles.  To  identify  overall  trends, 
the  numbers  of  journal  articles  per  year  in  this  area  were  examined.  A  ranked  list  of  authors  and 
institutions  was  created  from  the  residual  list  of  conference  articles  and  papers,  as  well  as  the 
number  of  conference  papers  written  per  country  and  per  year  for  comparison  with  the  trends  in 
the  journal  papers.  The  unfiltered  query  returned  6,126  results  (including  journal  articles  and 
papers,  conference  articles  and  papers,  conference  proceedings,  books,  monographs,  and 
dissertations,).  Of  the  6,126  total,  4,805  were  journal  articles,  and  about  1,200  were  conference 
papers.  The  remaining  120  were  a  mix  of  conference  proceedings,  books,  monographs, 
dissertations,  etc.  The  time  history  is  shown  in  figure  4.1. 


Figure  4.1  Comparison  of  Journal  and 
Conference  Articles  Related  to  Hypersonics 
(Source:  Ei  Compendex,  1991-2003) 


-•—JOURNAL 

ARTICLES 


CONFERENCE 
ARTICLES  & 
PAPERS 


Year 


Table  4.1  ranks  the  number  of  papers  published  by  country. 
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TABLE  4.1 :  NUMBER  OF  PAPERS  PER  COUNTRY 
Source:  Ei  Compendex,  1991-2004 


COUNTRY 

PAPERS 

USA 

1890 

JAPAN 

406 

CHINA 

347 

RUSSIA 

205 

ENGLAND 

200 

GERMANY 

175 

TAIWAN 

108 

INDIA 

86 

ITALY 

79 

FRANCE 

61 

CANADA 

59 

ISRAEL 

55 

SOUTH  KOREA 

53 

AUSTRALIA 

21 

NETHERLANDS 

20 

BRAZIL 

13 

GREECE 

11 

SWITZERLAND 

11 

SPAIN 

9 

POLAND 

8 

The  five  most  published  countries  included  the  United  States,  Japan,  China,  Russia,  and  England 
with  the  United  States  having  a  substantial  lead.  This  figure  shows  that  the  number  of  journal 
articles  published  decreased  from  approximately  460  in  1994  to  about  270  in  1998,  and  then 
resurged  to  about  430  in  2003.  This  resurgence  correlates  with  the  establishment  of  the  National 
Aerospace  Initiative  in  2001.  The  average  number  of  journal  papers  from  1991-2003  is  about 
362.  The  average  number  of  conference  papers  per  year  is  about  89. 
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Table  4.2  orders  the  24  most  published  organizations. 


TABLE  4.2:  TOP  24  INSTITUTIONS 

Source:  Ei  Compendex,  1991-2004 

INSTITUTION 

COUNTRY 

PAPERS 

NASA  LANGLEY  RESEARCH  CENT 

USA 

158 

NASA  AMES  RESEARCH  CENT 

USA 

59 

TOHOKU  UNIV,  JPN 

JAP 

40 

OLD  DOMINION  UNIV 

USA 

36 

BEIJING  UNIV  OF  AERONAUTICS  AND  ASTRONAUTICS 

CHINA 

34 

UNIV  OF  MARYLAND 

USA 

31 

VIRGINIA  POLYTECHNIC  INST 

USA 

30 

NASA  LEWIS  RESEARCH  CENT, 

USA 

27 

NORTH  CAROLINA  STATE  UNIV 

USA 

27 

STANFORD  UNIV 

USA 

27 

NORTHWESTERN  POLYTECHNICAL  UNIV,  CHINA 

USA 

25 

UNIV  OF  TEXAS  AT  AUSTIN 

USA 

25 

PRINCETON  UNIV 

USA 

24 

UNIV  OF  FLORIDA 

USA 

22 

PENN  STATE  UNIV 

USA 

20 

NATL  AEROSPACE  LAB,  TOKYO 

JAP 

19 

KYUSHU  UNIV 

JAP 

18 

NAGOYA  UNIV 

JAP 

18 

SANDIA  NATL  LAB,  ALBUQUERQUE 

USA 

18 

FLORIDA  STATE  UNIV 

USA 

17 

IOWA  STATE  UNIV 

USA 

17 

UNIV  OF  CAMBRIDGE 

USA 

17 

UNIV  OF  CINCINNATI 

USA 

16 

UNITED  TECHNOLOGIES  RESEARCH  CENT 

USA 

15 

The  U.S.  Federal  laboratories  represented  include  three  NASA  centers  and  the  Sandia  National 
Laboratories.  NASA  Langley  clearly  has  been  the  dominant  force  in  this  particular  area.  No 
DOD  laboratories  appear  in  the  “Top  24  Institutions.”  The  other  U.S.  institutions  are  all 
universities,  except  for  the  United  Technologies  Research  Center. 
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Table  4.3  orders  the  top  20  journals.  The  top  five  journals  listed  include  the  AIAA  Journal,  the 
Journal  of  Spacecraft  and  Rockets,  the  Journal  of  Propulsion  and  Power,  the  Journal  of  Aircraft, 
and  the  Journal  of  Thermophysics  and  Heat  Transfer.  All  of  these  are  published  by  the  AIAA. 


TABLE  4.3:  TOP  20  JOURNALS 
Source:  Ei  Compendex,  1991-2004 


JOURNAL  PAPERS 

AIAA  Journal  644 

JOURNAL  OF  SPACECRAFT  AND  ROCKETS  404 

JOURNAL  OF  PROPULSION  AND  POWER  355 

JOURNAL  OF  AIRCRAFT  219 

JOURNAL  OF  THERMOPHYSICS  HEAT  TRANSFER  164 

NIPPON  KIKAI  GAKKAI  RONBUNSHU,  B  HEN/TRANSACTI  132 

TUIJIN  JISHU/JOURNAL  OF  PROPULSION  TECHNOLOGY  108 

JOURNAL  OF  FLUID  MECHANICS  99 

IZV  AN  SSSR  MEKH  ZHIDK  GAZA  81 

COMPUT  FLUIDS  70 

INTERNATIONAL  JOURNAL  FOR  NUMERICAL  METHODS  IN  67 

J  GUID  CONTROL  DYN  65 

TRANSACTIONS  OF  THE  JAPAN  SOCIETY  FOR  AERONAUT  56 

TEPLOFIZIKA  VYSOKIKH  TEMPERATUR  55 

TRANS  JPN  SOC  AERONAUT  SPACE  SCI  55 

JOURNAL  OF  SOUND  AND  VIBRATION  53 

TEPLOFIZ  VYS  TEMP  52 

AERONAUTICAL  JOURNAL  49 


Figure  4.1  and  tables  4.1  through  4.3  provide  information  regarding  worldwide  effort  in  the  area 
of  hypersonic  technology  and  identify  the  centers  of  activity  in  the  United  States  (at  least  in  so 
far  as  they  participate  in  the  open  literature).  This  material  would  be  useful  to  and  provides 
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necessary  background  for  an  EWA.  However,  it  is  far  too  general  in  character  to  form  the  basis 
of  an  EWA.  The  reason  for  this  relates  to  the  large  number  of  distinctly  different  scientific  and 
technical  disciplines  encompassed  by  the  descriptor  “Hypersonic  Technology.”  If  one  attempted 
an  assessment  it  would  be  extraordinarily  difficult  to  identify  and  assemble  a  group  of  experts 
who  were  qualified  to  collectively  provide  a  substantive  assessment  of  the  area.  For  example,  the 
NRC  study  on  the  NAI  noted  that  hypersonic  technology  involves  several  critical  technologies. 

These  include: 

•  Propulsion 

•  Thermal  environment  prediction,  protection  and  management 

•  Integrated  airframe  structures  and  cryogenic  tanks 

•  Vehicle  design,  optimization  and  simulation 

We  will  refer  to  the  above  as  Level  2  descriptors  of  CESTs  and  “hypersonic  technology”  as  a 
Level  1  descriptor.  Further  examination  demonstrates  that  each  of  these  Level  2  critical 
technologies  subdivides  into  additional  critical  technologies.  For  example,  Bowcutt 
(“Hypersonic  Technology  Status  and  Roadmap”,  Presentation  to  AIAA  HyTASP  Program 
Committee,  Dec.  18,  2003,  http://www.aiaa.org/Participate/Uploads/TFAB%20HyTASP.pdf) 
showed  that  the  Level  2  “Propulsion”  CEST  subdivides  into  Level  3  CESTs  as  follows: 

•  Engine  materials 

•  Cooled  engine  panels 

•  Scramjet  combustors 

•  Fuel  injectors/Flame  holders 

•  Integrated  flow-path-hydrogen 

•  Integrated  flow-path-hydrocarbon 

•  Engine  seals 

•  Engine  active  control 

•  Turbine-Based  Combined  Cycle  (TBCC)  hypersonic  engine  flow-path  integration 

One  can  further  subdivide  the  above  list  of  Level  3  descriptors  into  Level  4  CEST  issues,  and  so 
forth.  This  illustrates  the  difficulty  of  conducting  a  conventional  assessment  of  the  total  DOD 
S&T  program.  Each  of  the  subdivisions  involves  different  disciplines  and  different  communities 
of  expertise.  The  number  of  experts  that  one  would  need  to  collect  together  for  a  conventional 
assessment  is  just  unreasonable.  As  discussed  in  the  previous  section,  this  proliferation  of 
disciplines  presents  a  difficult  problem  even  for  a  focused  review  of  a  small  subset  of  the  DOD 
S&T  program.  It  presents  an  insurmountable  problem  for  a  conventional  review  approach  to  an 
EWA  of  the  entire  DOD  S&T  program.  However,  the  next  level  (Level  4)  subdivision 
approaches  the  level  at  which  scientists  and  engineers  actually  work.  This  may  present  an 
opportunity  for  a  non-conventional  approach  by  exploitation  of  the  fact  that  Level  4  approaches 
the  point  where  the  various  subject  matter  experts  can  usually  quickly  make  informed  judgments 
regarding  the  state  of  the  art  and  the  prognosis  for  progress  in  S&T  with  which  they  are  very 
familiar.  This,  however,  places  significant  constraints  on  the  documentation  of  the  program.  To 
explore  this  point  we  will  examine  the  Level  4  detail  for  two  of  the  CESTs  identified  by 
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Bowcutt.  In  particular,  we  will  discuss  in  some  detail  “Turbine-Based  Combined  Cycle  (TBCC) 
hypersonic  engine  flow-path”  and  in  lesser  detail  “Engine  materials”. 

Future  hypersonic  air-breathing  UCAVs  for  time-critical  theater  operations,  and  reusable  launch 
vehicles  for  prompt  global  response  and  routine  space  access,  will  require  2-3  propulsion  modes 
to  operate  across  their  design  speed  ranges.  One  of  the  most  likely  propulsion  systems  for  such 
applications  will  be  the  TBCC  propulsion  plant,  which  consists  of  a  supersonic  (-Mach  4) 
turbine  engine  closely  integrated  with  a  dual-mode  ramjet/scramjet  engine.  The  study  authors 
worked  with  Kevin  Bowcutt  of  the  Boeing  Company  to  construct  a  hypothetical  (but  technically 
realistic)  program  that  would  address  the  key  issues  associated  with  the  TBCC.  It  was  found  that 
it  will  be  necessary  to  develop  a  TBCC  engine  inlet  system  that  efficiently  delivers  air  flow 
required  by  the  turbine  engine  when  operating  alone,  air  flow  required  by  the  scramjet  when 
operating  alone  and  to  both  engines  simultaneously  during  the  transition  from  turbine  to 
scramjet.  It  is  also  necessary  to  maintain  inlet  fluid  dynamic  stability  during  mode  transition  and 
in  all  other  operational  engine  modes.  One  must  develop  a  TBCC  engine  nozzle  system  that 
efficiently  expands  flow  to  produce  thrust  for  the  turbine  engine  when  operating  alone,  for  the 
scramjet  engine  when  operating  alone,  and  for  both  engines  when  operating  simultaneously 
during  mode  transition.  It  is  necessary  to  develop  an  aircraft  and  engine  flowpath  control  system 
that  adequately  stabilizes  and  controls  the  aircraft  and  TBCC  engine  flowpaths  during  engine 
mode  transition  and  all  other  flight  phases.  Lastly,  it  will  be  necessary  to  develop  the  material, 
structural,  and  mechanical  design  for  the  articulated  TBCC  inlet  and  nozzle  systems,  and  a 
thennal  management  system  for  the  turbine  engine  when  it  is  shutdown  but  exposed  to  scramjet 
flowpath  heating.  In  the  following  paragraphs  we  will  outline  the  resultant  hypothetical  program 
aimed  at  achieving  these  objectives.  One  of  the  goals  here  is  to  illustrate  how  one  might 
articulate  the  CESTs  for  a  program  of  this  nature  and  how  one  might  document  the  program  so 
as  to  be  useful  to  an  EWA. 

A  reasonable  approach  to  the  above  objectives  would  be  to  define  TBCC  inlet  and  nozzle 
geometry  and  articulation  requirements  based  initially  upon  knowledge  of  existing  supersonic 
inlets  and  nozzles,  empirical  and  theoretical  design  guidelines  and  operability  (i.e.,  stability) 
criteria,  engine  perfonnance  characteristics,  and  analysis  results  derived  from  engineering  codes 
and  computational  fluid  dynamics  (CFD).  This  would  be  followed  by  building  sub-scale  inlet 
and  nozzle  models  and  testing  them  in  wind  tunnels,  both  statically  and  with  dynamic 
articulation  of  the  variable  geometry  components.  One  could  then  use  the  knowledge  gained 
from  testing  and  analysis  to  optimize  inlet  and  nozzle  designs.  This  would  be  aided  by 
mathematical  optimization  and  multidisciplinary  design  optimization  techniques. 

It  would  also  be  necessary  to  develop  inlet  and  nozzle  control  algorithms,  and  then  test  them  via 
dynamic  simulation  using  data  derived  from  testing  and  CFD  analysis.  Vehicle  stability  and 
control  during  engine  mode  transition  would  be  verified  via  dynamic  simulation  using  CFD- 
derived  aero  data  in  conjunction  with  inlet  and  nozzle  test  data.  Wind  tunnel  tests  would  be 
conducted  of  an  aircraft  model  with  variable  inlet  to  verify  inlet  stability  during  mode  transition, 
and  aero  data  would  be  gathered  to  verify  vehicle  stability  and  control  during  the  transition 
event. 
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The  program  would  design  the  structure  and  mechanisms,  including  high-temperature  seals,  for  a 
full-scale  TBCC  inlet  and  nozzle,  and  then  bench  test  critical  inlet  and  nozzle  components  to 
verify  predicted  operation  and  perfonnance.  It  would  also  design  a  thermal  management  system 
(TMS)  for  the  turbine  engine  flowpath  and  analyze  it  using  fluid,  thennal,  and  structural  analysis 
codes,  and  then  bench  test  critical  TMS  components  to  verify  predicted  operation  and 
performance. 

The  program  outlined  above  involves  several  Level  4  CESTs  that  must  be  resolved  for  the  total 
program  to  be  successful.  The  prospects  for  conducting  an  EWA  would  be  enhanced  if  the 
CESTs  could  be  articulated  such  that  experts  in  each  CEST  area  could  quickly  comment  on  the 
likelihood  of  the  CESTs  being  advanced  in  accordance  with  the  stated  program.  It  is  suggested 
that  the  following  articulation  of  the  CESTs  would  suffice. 

1)  Hypersonic  engine  seals  for  articulated  (i.e.,  variable  geometry)  inlet  and  nozzle  structural 
elements. 

2)  A  dual-flow-path  TBCC  hypersonic  engine  inlet  that  compresses  and  feeds  air  to  the  turbine 
and  scramjet  engines,  both  separately  and  concurrently.  This  entails  complex  variable  geometry, 
high  compression  efficiency  and  stable  operation  for  both  the  turbine  and  scramjet  flow-paths 
operating  separately  and  simultaneously. 

3)  A  dual-flow-path  hypersonic  engine  nozzle  that  expands  exhaust  gases  from  the  turbine  and 
scramjet  engines,  both  separately  and  simultaneously.  This  entails  complex  variable  geometry  of 
hot  structures  and  high  expansion  efficiency  for  both  engine  flow-paths  operating  separately 
and  simultaneously. 

4)  A  post-shutdown  hypersonic  turbine  engine  thermal  management  system  (TMS)  that 
maintains  turbine  material  temperature  limits  when  the  turbine  is  shut  down  and  the  scramjet  is 
operating  adjacent  to  the  turbine  flow-path. 

5)  A  hypersonic  engine  flow-path  control  system  and  associated  logic  and  algorithms  (i.e., 
software)  that  maintain  stable  engine  flow-path  operation  and  thrust  performance,  and  concurrent 
aircraft  stability  and  control  during  TBCC  mode  transition  from  ramjet  to  scramjet  operation,  and 
vice  versa. 

The  program  with  the  CESTs  outlined  above  becomes  complete  when  milestones,  metrics,  and 
funding  are  assigned  to  each  CEST.  At  that  point  the  program  can  be  assessed  regarding  the 
realism  of  the  milestones,  metrics,  and  funding.  Continuing  with  the  above  hypothetical  example, 
we  propose  the  milestones,  metrics  and  funding  for  each  CEST  illustrated  below.  We  employ  a 
format  similar  to  that  used  by  the  NASA  Technology  Inventory  (see  Appendix  C),  and  have 
combined  CESTs  2  and  3. 
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TABLE  4.4  CEST  1:  Hypersonic  engine  seals  (U) 


Technology  Metrics: 


Metric 

Metric  Unit 

SOA 

Planned  Value 

Maximum  use 
temperature 

Temperature  °  F 

1400°  F  (metallic) 
1800°  F  (ceramic) 

1800°  F  (metallic) 

2500-3000°  F  (ceramic) 

Durability/life 

#  of  cycles  before 
replacement 

1000’s  for  metallic 
10-  100  for 
ceramic 

1000’s  for  <2000°  F 

100’s  at  2500°  F 

10’s  at  3000°  F 

Milestones: 


TRL 

Year 

Description 

6 

2008 

Design  and  fabricate  seals  and 
complete  critical  component  testing 
in  a  relevant  environment 

‘Current  TRL  =  2-3 
Funding: 


Fiscal  Year 

2005 

2006 

2007 

2008 

Funding 

Profile 

$1200K 

$1500K 

$1800K 

$1500K 

Technical  Work  Years 


Fiscal  Year 

2005 

2006 

2007 

2008 

Technical  Work 

Years 

3 

4 

4 

3 

Suggested  Experts: 


Last  Name* 

First  Name* 

Organization 

Email 

Phone 

NASA  Glenn 
Research  Center 

NASA  Glenn 
Research  Center 

‘names  deleted 
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Table  4.5  CEST  2:  Dual  flow-path  TBCC  hypersonic  engine  inlets  and 
nozzles  (U) 

Technology  Metrics: 


Metric 

Metric  Unit 

SOA 

Planned  Value 

Integrated  dual  inlet 
operability 

#  of  inlets  operating 
simultaneously  at  high 
performance 

1 

2  at  SOA  inlet  recovery 
and  operability/stability 

Integrated  dual  nozzle 
operability 

#  of  nozzle  operating 
simultaneously  at  high 
performance 

1 

2  at  SOA  thrust 
coefficient 

Milestones: 


TRL 

Year 

Description 

3 

2006 

TBCC  inlet  and  nozzle  design  and 
analysis 

4 

2007 

Wind  tunnel  test  sub-scale  TBCC  inlet 
and  nozzle 

5 

2008 

Update  inlet  and  nozzle  designs,  and 
tunnel  test  inlet  on  a  sub-scale  aircraft 
model  with  an  articulated  inlet 

‘Current  TRL  =  2-3 
Funding: 


Fiscal  Year 

2005 

2006 

2007 

2008 

Funding  Profile 

0 

S3000K 

S4000K 

S3000K 

Technical  Work  Years 


Fiscal  Year 

2005 

2006 

2007 

2008 

Technical  Work  Years 

0 

8 

8 

8 

Suggested  Experts: 


Last  Name* 

First  Name* 

Organization 

Email 

Phone 

Johns  Hopkins  Applied  Physics  Laboratory 

AF  Research  Laboratory/Propulsion  Directorate 

NASA  Langley  (Retired) 

TechLand  Research,  Inc. 

NASA  Langley  (Retired) 

names  deleted 
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Table  4.6  .  CEST  3:  High  -speed  turbine  engine  thermal  management  system  (TMS);  for  turbine 
engine  during  operation  and  when  shutdown/cocooned  during  scramjet  operation 


Technology  Metrics: 


Metric 

Metric  Unit 

SOA 

Planned  Value 

Maximum  turbine 
engine  soak 
temperature 

Temperature 

160-200 " 

F 

Maximum  of  350-600°  F;  optimum  value  must  be 
determined  by  trade  study  (limited  by  fuel  coking, 
lubricants,  seals,  engine  structure,  electronics) 

TMS  system 
weight 

Weight 

No  impact  to  engine  thrust-to-weight  via  highly 
integrated  turbine  and  scramjet  TMS 

Milestones: 


TRL 

Year 

Description 

2 

2006 

Conduct  trade  study  to  establish  optimum  turbine  engine  soak  temperature,  and 
create  high-temperature  engine  component  technology  development  roadmap  (which 
become  additional  TBCC  critical  enabling  technologies) 

3 

2007 

TMS  design  and  analysis  complete 

4 

2008 

Critical  TMS  component  bench  testing  and  analysis  update 

‘Current  TRL  =  2-3 
Funding: 


Fiscal  Year 

2005 

2006 

2007 

2008 

Funding  Profile 

N/A 

S1000K 

S2000K 

S3000K 

Technical  Work  Years 


Fiscal  Year 

2005 

2006 

2007 

20080 

Technical  Work  Years 

0 

3 

4 

5 

Suggested  Experts 


Last  Name* 

First  Name* 

Organization 

Pratt  and  Whitney 

Pratt  and  Whitney 

Pratt  and  Whitney 

Air  Force  Research  Lab 

Pratt  and  Whitney,  retired 

names  deleted 


Table  4.7.  CEST  4:  Hypersonic  Engine  Control  System  Development 
Technology  Metrics: 


Metric 

Metric  Unit 

SOA 

Planned  Value 

Inlet  stability  during 
engine  mode 
transition 

Stability  margin 

N/A 

5% 

Aerodynamic 
dynamic  stability 
during  engine  mode 
transition 

Control  system  phase 
and  gain 

N/A 

6  db  gain  and  45° 
phase  margins 

Milestones: 


TRL 

Year 

Description 

3 

2007 

Develop  inlet,  nozzle  and  aircraft  control  laws  using  aerodynamic  propulsion 
analysis  of  same.  Perform  dynamic  simulations. 

4 

2008 

Use  inlet  control  laws  in  wind  tunnel  test  of  integrated  inlet/airframe.  Use 
tunnel  data  to  update  control  laws  and  simulations. 

*Current  TRL=2.3 
Funding 


Fiscal  Year 

2005 

2006 

2007 

2008 

Funding 

$1000k 

$1500k 

$1500k 

S2000k 

Technical  Work  Years 


Fiscal  Year 

2005 

2006 

2007 

2008 

Technical  Work  Years 

3 

3 

3 

4 

Suggested  Experts: 


Last  Name* 

First  Name* 

Organization 

Email 

Phone 

Boeing,  Huntington  Beach 

NASA,  Dryden 

Air  Force  Research  Laboratory 

Boeing,  St.  Louis 

*names  deleted 


The  above  examples  suggest  a  documentation  fonnat  for  a  DOD  S&T  work  unit.  The  basic 
theme  behind  the  format  is  the  careful  identification  by  the  principal  investigators  of  all  the 
CESTs  that  the  work  unit  is  funded  to  address.  The  active  involvement  of  the  principal 
investigators  in  preparing  the  proper  documentation  is  essential.  They  are  the  individuals  who 
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best  know  the  program.  In  this  regard  it  should  be  noted  that  the  above  examples  have  included  a 
list  of  recommended  assessors.  The  principal  investigator  should  provide  such  a  list  as  part  of  the 
program  documentation.  An  outline  of  the  suggested  format  is  presented  and  table  4.8.  If  the 
entire  DOD  S&T  program  were  documented  as  shown  in  table  4.8,  the  resulting  database  might 
form  the  basis  for  an  EWA.  The  logical  custodian  for  this  database  is  DTIC.  For  purpose  of 
illustration,  Appendix  A  provides  a  completed  version  of  table  4.8  for  the  hypersonic  example 
summarized  in  tables  4.4  -  4.7. 


Table  4.8  :  Suggested  Work  Unit  documentation  format 


Work  Unit  Summary  Form  XX-XXXX 


1.  Title:  2.  Date  of  Summary: 

3.  Responsible  Organization 


a.  Name:  b.  Address: 

c.  Principle  Investigator: 


Name 

Phone 

Email  Address 

4.  Classification  Level: 

a:  Summary  Level:  b:  Work  Level: 

5.  Military  need: 

6.  Technical  Objective: 

7.  Progress: 

8.  Critical  Enabling  Science  and  Technology  (CEST) 
a:  CEST  1: 


Approach 
Technology  Metrics 


Metric 

Metric  Unit 

SOA 

Planned  Value 

Milestones 


TRL 

Year 

Description 

*Current  TRL  = 

Page  1 


Funding 


Fiscal  Year 

2005 

2006 

2007 

2008 

2009 

Funding 

Profile 

Technical  Work  Years 


Fiscal  Year 

Technical 

Work  Years 

9.  Suggested  Experts: 


Last  Name 

First  Name 

Organization 

Email 

Phone 

Click  here  to  add  an 
additional  CEST 

10.  Total  Funding: 


Fiscal  Year 

Total 

Funding 

11.  Program  Participants: 


Contracts  with  whom 

Grants  with  whom 

Government  organization 

12.  DoD  Subject  Areas:  http://www.dtic.mil/dtic/subcatquide/ 

13.  Key  Words: 

Paee2 


Table  4.8  has  been  streamlined  so  as  to  highlight  those  elements  that  are  most  needed  for  an 
EWA.  The  table  would  ultimately  incorporate  additional  fields  (e.g.,  program  element  numbers) 
to  allow  traceability  to  appropriations,  etc.  The  database  that  would  result  would  be  useful 
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independent  of  an  EWA.  The  NASA  Technology  Inventory  (NTI)  database  provides  a  good 
illustration  of  such  utility.  This  point  is  discussed  in  Appendix  C,  which  provides  a  brief 
description  of  the  NTI.  It  also  provides  several  displays  that  this  study  was  able  to  easily 
construct  from  the  online  NTI.  These  displays  are  intended  to  show  that  the  existence  of  this 
database  and  its  functionality  allow  authorized  users  to  inquire  about  the  NASA  Technical 
Program  from  the  highest  level  of  aggregation  down  to  the  individual  work  units  with  the  click 
of  a  mouse.  This  functionality  could  and  should  be  available  to  DOD  decision  makers  and 
performers  regarding  the  DOD  S&T  program. 

Establishing  Communities  of  Experts 

The  following  paragraphs  will  outline  an  approach  to  identifying  the  needed  communities  of 
experts.  While  the  establishment  of  these  communities  of  experts  is  a  critical  aspect  of  the  EWA 
methodology  discussed  in  this  report,  it  is  also  in  the  best  interests  of  DOD  independent  of  an 
EWA.  Science  and  technology  will  continue  a  trend  towards  increased  sophistication  and 
complexity.  Keeping  current  in  these  developments  will  become  increasingly  difficult.  This  will 
be  exacerbated  by  the  inevitable  trend  toward  globalization  of  S&T.  In  order  to  cope  with  these 
developments,  DOD  must  reach  out  in  a  constructive  fashion  to  engage  the  larger  U.S.  scientific 
and  technical  communities  regarding  S&T  for  national  security.  This  must  be  done  within  the 
context  of  a  national  purpose  rather  than  through  some  competitive  free  market  process.  It  is 
unfortunate  that  the  constructive  engagement  between  the  DOD  and  the  U.S.  scientific 
community  has  greatly  diminished  over  the  past  thirty  years.  This  situation  needs  to  be  rectified. 
Most  American  scientists  and  engineers  are  well  aware  that  the  nation’s  long-tenn  strength 
(military  and  economic)  depends  upon  the  federal  government’s  stewardship  of  the  nation’s  long 
tenn  S&T  investments.  Many  of  these  scientists  and  engineers  would  be  honored  to  be 
considered  members  of  a  community  of  experts  that  is  contributing  to  this  national  purpose.  A 
public  policy  to  accomplish  this  should  be  established,  and  a  public  relations  program  should  be 
mounted  to  articulate  this  policy  to  the  nation’s  scientific  and  technical  communities.  If 
successfully  done  it  would  be  therapeutic  for  the  S&T  community  and  would  provide  the  type  of 
access  to  this  community  that  can  be  achieved  only  by  tapping  into  people’s  commitment  to  the 
nation’s  health  and  wellbeing.  Such  an  undertaking  would  be  a  strong  motivator  for  individual 
scientists  and  engineers  to  participate  in  the  EWA  discussed  below. 

Tables  4.4  through  4.7  reduce  the  program  to  a  succinct  statement  of  CESTs,  metrics, 
milestones,  funding,  and  technical  work  years.  A  database  of  such  tables  for  the  entire  DOD 
S&T  program  might  form  the  basis  for  an  EWA.  However,  in  order  to  do  this  the  program 
documentation  must  enable  the  finding  of  appropriate  assessors.  One  would  like  to  identify  a 
large  number  of  assessors  in  order  to  gain  adequate  scientific  and  technical  coverage  to  avoid 
some  of  the  “small  sample”  problems  that  often  occur  in  the  conventional  review  process.  The 
CESTs  may  provide  the  key  to  doing  this.  For  example,  if  the  CESTs  are  properly  articulated 
then,  by  employing  available  databases  and  search  engines,  the  tables  might  form  the  basis  for 
identifying  a  pool  of  active  scientists  and  engineers  from  which  a  community  of  experts  could  be 
formed  by.  It  is  important  to  understand  that  each  CEST  will  have  its  own  community  of  experts. 
We  will  apply  this  line  of  reasoning  to  the  CESTs  identified  above.  The  first  was  hypersonic 
engine  seals  for  articulated  inlet  and  nozzle  structural  elements.  The  broad  search  of  the  field  of 
hypersonic  technology  reported  in  table  4.3  indicates  that  the  AIAA  journals  have  published  the 
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largest  number  of  papers  in  the  area  of  hypersonic  technology.  It  would,  therefore,  seem 
reasonable  to  query  the  AIAA  Electronic  Library  (http://www.aiaa. org/content.cfm?pageid=406) 
regarding  publications  in  the  area  of  hypersonic  engine  seals  in  order  to  identify  individuals  who 
publish  in  this  area.  A  query  on  “hypersonic  engine  seals”  for  the  period  1995  through  2005 
found  25  records  that  identified  20  U.S.  authors  along  with  their  affiliated  institutions.  The  same 
query  of  the  Google  search  engine  identified  a  number  of  sites  including  one  that  provides  the 
proceedings  of  the  “2002  NASA  Seal/  Secondary  Air  System  Workshop.”  A  simple  search  of  the 
proceedings  table  of  contents  identified  17  additional  U.S.  authors  along  with  their  institutions.  A 
more  sophisticated  search  using  the  Google  Scholar  search  engine  with  the  same  query  for  the 
same  timeframe  yielded  103  U.S.  authors  along  with  their  affiliations.  It  is  interesting  to  note  that 
the  Google  Scholar  search  identified  papers  in  the  AIAA  Electronic  library  that  were  not 
identified  by  the  direct  search  of  the  AIAA  Electronic  Library.  Also,  the  direct  search  of  the 
AIAA  Electronic  library  found  some  papers  not  identified  by  Google  Scholar.  The  total  number 
of  individuals  identified  who  had  U.S.  affiliations  was  129.  A  complete  listing  of  these 
individual’s  organization  will  be  found  in  Appendix  B,  table  Bl.  The  names  of  the  individuals 
are  not  listed  for  privacy  reasons.  This  group  could  be  considered  to  be  a  potential  community  of 
experts  for  the  CEST  “Hypersonic  engine  seals.”  The  time  that  it  took  to  generate  the  input  for 
the  entire  listing  of  the  potential  community  of  experts  for  hypersonic  engine  seals  was  less  than 
a  second  (plus  perhaps  another  15  minutes  of  manual  work  to  eliminate  individuals  who  were 
clearly  inappropriate).  The  search  engines  used  above  did  not  offer  the  option  of  automatically 
extracting  all  the  authors  and  their  affiliations.  This  had  to  be  accomplished  manually  and  was 
unacceptably  time  consuming  for  an  EWA.  However,  since  the  author  names  and  affiliations  are 
machine  readable,  it  should  be  straightforward  to  automate  this  extraction  process.  Once  this  is 
done,  arriving  at  the  potential  community  of  experts  for  each  CEST  could  be  totally  automated 
and  performed  in  a  matter  of  seconds. 

A  large  number  of  search  engines  are  available  for  scientific  and  technical  inquiries.  Some  can 
be  utilized  at  no  cost,  while  others  can  be  quite  costly.  The  latter  search  engines  are  best  suited 
when  a  very  complete  analysis  of  an  area  is  desired.  However,  the  establishment  of  a  potential 
community  of  experts  in  an  area  does  not  require  a  complete  analysis  of  all  work  that  has  been 
done  in  that  area.  It  requires  only  that  a  large  enough  sample  be  found  of  individuals  who  have 
published  recently  in  the  area  in  question.  In  the  cases  examined  in  this  study  it  was  found  that 
the  free  search  engines,  such  as  Google  Scholar,  provided  satisfactory  potential  communities  of 
experts.  It  is  expected  that,  in  the  coming  years,  these  free  search  engines  will  improve 
considerably  and  that  the  ability  to  find  satisfactory  potential  communities  of  experts  rapidly  and 
at  no  cost  will  improve  accordingly. 

We  have  used  the  term  “potential  community  of  experts”  because  the  listing  in  table  Bl  contains 
individuals  who  will  be  well  matched  to  assessing  the  program  outlined  in  table  4.4  and  others 
who  will  not  be  well  matched.  One  could  narrow  down  the  community  of  experts  by  several 
methods.  At  one  extreme  one  could  proceed  by  asking  the  principal  investigator  for  a  particular 
CEST  to  identify  from  table  Bl  those  individuals  best  suited  to  perfonn  an  assessment  (this 
request  could  be  automatically  prompted  upon  the  PI  providing  documentation  of  the  program). 
At  the  other  extreme  one  could  interrogate  databases  such  as  the  Community  of  Science 
Expertise  database  (http://expertise.cos.com/about/expertise.shtml)  for  each  individual  listed  in 
table  Bl  in  order  to  find  the  best  matches  for  the  desired  assessment.  At  the  current  state  of 
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technology  this  latter  approach  likely  would  involve  an  unacceptable  level  of  human 
intervention.  An  intermediate  approach  would  be  to  have  the  principal  investigator  responsible 
for  a  particular  CEST  identify,  in  his/her  formal  documentation,  a  number  of  individuals 
believed  to  be  appropriate  assessors  for  the  CEST  (the  “suggested  community  of  experts”  shown 
in  tables  4.4  through  4.7)  and  have  the  principal  investigator,  prior  to  submitting  the 
documentation,  gain  their  agreement  to  participate  in  an  assessment.  One  could  then 
automatically  provide  these  individuals  a  list  such  as  shown  in  table  B1  (with  the  names 
included)  and  ask  them  to  identify  individuals  from  that  list  whom  they  believe  are  well  suited  to 
conduct  an  assessment.  This  approach  has  the  benefit  of  putting  some  distance  between  the  PI 
and  the  assessors  and  having  the  assessors  selected  by  experts.  Another  approach  would  be  to  use 
the  full  list  given  in  table  B 1  and  have  the  members  of  the  potential  community  of  experts  decide 
as  individuals  whether  or  not  they  are  appropriate  to  participate  in  the  assessment. 

The  same  approach  used  to  construct  table  B1  was  used  to  identify  potential  communities  of 
experts  for  the  other  CESTs  identified  in  tables  4.5  through  4.7.  In  these  cases  the  searches  were 
limited  to  Google  Scholar.  In  case  of  the  CEST  “Dual  flow  path  TBCC  engine  inlets  and 
nozzles,”  a  potential  community  of  experts  of  28  individuals  was  found.  For  the  CEST  “Post 
shutdown  turbine  engine  thermal  management  System  (TMS)”  a  potential  community  of  experts 
consisting  of  70  individuals  was  found.  For  the  CEST  “Hypersonic  Engine  Control  System 
Development”  a  potential  community  of  experts  consisting  of  134  individuals  was  found.  The 
names  and  affiliations  of  the  potential  communities  of  experts  will  be  found  in  Appendix  B, 
tables  B2  through  B4.  In  each  of  the  above  cases  a  potential  community  of  experts  was  found 
that  is  large  enough  to  suggest  that  meaningful  statistics  could  emerge  if  a  reasonable  fraction 
(say  50  percent)  of  the  potential  community  of  experts  agrees  to  participate  in  the  assessment.  In 
general,  a  lower  limit  of  about  10  would  need  to  be  placed  on  the  size  of  the  final  community  of 
experts  in  order  to  obtain  meaningful  statistics. 

The  turbine  based  combined  cycle  (TBBC)  engine  flow  path  example  discussed  above  is  typical 
in  character  of  many  of  DOD  programs  funded  at  the  advanced  6.2  and  6.3  stage.  Such  programs 
often  have  a  systems  character  to  them.  Our  study  finds  that,  if  the  CESTs  are  properly  specified, 
it  is  straightforward  to  find  a  potential  community  of  experts  for  the  various  CESTs.  The 
searches  we  conducted  produced  10s  to  100  or  more  potential  members  for  the  community  of 
experts  associated  with  a  particular  CEST.  These  searches  were  all  performed  in  less  than  one 
second  of  search  time.  The  DOD  S&T  program,  however,  also  supports  a  substantial  amount  of 
work  that  can  be  generic  in  character  and  is  often  funded  at  the  6.1  and  early  6.2  levels.  For 
example,  work  on  materials  as  often  in  this  category.  This  work  may  investigate  materials 
properties  with  the  objective  of  identifying  material  systems  that  might  be  useful  in  some  of  the 
stressful  environments  encountered  by  DOD  systems.  This  study  has  found  that  the  basic 
approach  outlined  for  the  TBBC  example  is  applicable  to  the  more  generic  S&T.  The  TRL 
assignment  is  not  particularly  useful  here  because,  for  these  programs,  it  usually  stays  in  1-2 
range  for  the  program  duration.  However,  the  idea  of  identifying  the  critical  S&T  issues  being 
addressed  by  these  programs  remains  valid.  The  concepts  of  the  state  of  the  art  and  planned 
advances  in  the  state  of  the  art  remain  valid,  as  do  the  concepts  of  milestones  and  communities  of 
experts.  For  these  programs,  the  importance  of  clearly  identifying  the  CEST  is  also  found  to  be 
key.  In  order  to  demonstrate  some  of  these  points  we  will  consider  briefly  the  topic  of  hypersonic 
engine  materials.  Furthermore,  to  illustrate  the  variety  of  search  engines  that  are  readily 
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available,  we  will  use  results  obtained  from  the  SCIRUS  search  engine 
(http://www.scirus.com/srsapp/).  Because  of  the  high  temperatures  encountered  in  hypersonic 
propulsion,  ceramic  matrix  composites  have  been  of  great  interest.  If  we  declare  “ceramic  matrix 
composites”  as  a  CEST  then,  for  the  period  1995-2005,  SCIRUS  identifies  1635  records  for  a 
full  document  search.  An  examination  of  these  records  shows  that  the  description  “ceramic 
matrix  composites”  covers  a  large  number  quite  different  material  systems  and  investigations.  It 
would  be  difficult  to  easily  extract  a  viable  potential  community  of  experts  from  this  broad 
listing.  Therefore,  “ceramic  matrix  composites”  is  a  Level  1  designation  and  is  not  a  satisfactory 
CEST.  One  must  inquire  at  a  more  specific  level.  A  significant  subfield  of  ceramic  matrix 
composites  is  that  of  fiber  reinforced  ceramic  matrix  composites.  A  full  document  search  at  this 
level  and  returns  602  records.  Examination  of  these  records  once  again  demonstrates  many 
different  material  systems  and  investigations  are  covered  by  this  level  of  description.  This  is  a 
Level  2  designation  and  is  not  a  satisfactory  CEST.  A  scientist  or  engineer  will  typically  be 
working  on  a  particular  material  system  such  as  silicon  carbide  fiber  reinforced  silicon  carbide 
(referred  to  by  the  community  as  SiC/SiC).  A  title  search  on  “SiC/SiC”  for  the  period  1995-2005 
identified  112  files.  While  this  is  approaching  the  level  where  an  assessment  might  be  done,  it  is 
actually  a  level  3  designation.  Even  within  this  narrow  field,  scientists  and  engineers  will  be 
working  on  some  particular  critical  issue,  such  as  the  interphase,  that  transitions  the  silicon 
carbide  fiber  to  the  silicon  carbide  matrix.  A  title  search  on  “SiC/SiC  interphase”  returns  39 
records.  This  is  a  Level  4  designation.  If  one  undertakes  to  identify  a  community  of  experts  at 
this  level,  then  one  should  find  individuals  who  could  quickly  and  authoritatively  comment  on  a 
DOD  S&T  program  that  is  conducting  working  in  this  narrow  area.  We  suggest  that  this  is  the 
proper  level  at  which  to  identify  the  CEST  (e.g.  “SiC/SiC  interphase”  for  this  example). 

The  results  from  a  search  to  identify  a  potential  community  of  experts  for  a  particular  CEST  can 
be  very  dependent  on  how  the  CEST  is  stated.  For  example,  when  the  term  SiC/SiC  in  the  above 
mentioned  title  search  was  replaced  with  its  full  name  “silicon  carbide  fiber  reinforced  silicon 
carbide”  then  the  SCIRUS  search  returned  only  9  records  instead  of  the  112  returned  for  the 
abbreviated  name.  The  reason  for  this  is  that  the  community  that  works  in  this  area  has  embraced 
the  designation  “SiC/SiC”  in  their  literature.  The  current  search  engines  are  not  sophisticated 
enough  to  figure  this  out.  One  must,  therefore,  not  only  define  the  CEST  at  the  proper  level  but 
must  also  properly  name  the  CEST.  The  individuals  best  qualified  to  do  this  identification  of  the 
CESTs  are  the  principal  investigators  for  the  various  projects.  The  principal  investigators  should 
be  required  to  identify  the  CESTs  as  part  of  the  formal  documentation  of  the  program.  They 
should  also  be  required  to  conduct  searches  on  their  proposed  CESTs  to  verify  that  their 
definitions  identify  an  adequate  number  (10-100)  of  potential  members  of  a  community  of 
experts  for  each  CEST  and  that  the  listing  they  are  finding  includes  names  that  they  would 
expect  including  some  of  those  they  have  identified  as  “suggested  experts.”  They  will  likely  need 
to  iterate  the  naming  of  the  CEST  several  times  to  get  it  right.  This  should  not  be  burdensome, 
because  each  search  typically  takes  less  than  a  second  to  accomplish.  The  proper  identification  of 
the  CESTs  must  be  included  in  the  fonnal  documentation  if  there  is  to  be  any  hope  of  conducting 
an  EWA.  This  is  because  the  exact  statement  of  the  CEST  will  form  the  basis  for  identifying  the 
individuals  who  will  be  asked  to  perform  the  assessment.  If  the  CEST  is  not  properly  identified, 
then  a  valid  community  of  experts  will  not  be  found.  It  would  be  helpful  to  get  feedback  from  the 
expert  assessors  regarding  whether  the  CEST  was  properly  identified/stated. 
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Assessment  Process 


In  this  section  the  enterprise-wide  assessment  methodology  is  described  sequentially.  Five  major 
steps  are  envisaged  including  the  preparation  of  Work  Unit  Summary  data  sheets,  extraction  of 
CESTs  by  DTIC  and  search  for  potential  reviewers,  selection  of  assessors  from  the  list  of 
potential  reviewers,  program  assessment  by  the  review  board  and  analysis  and  reports  of  the 
assessments.  The  two  highest  risk  elements  in  the  execution  of  this  proposed  process  are  the 
formulation  of  the  CESTs  in  the  Work  Unit  Summaries  at  the  appropriate  level  and  the 
generation  of  a  community  of  experts  using  advanced  search  engine  technology.  The  sequence  is 
as  follows: 

1.  The  PI  of  the  responsible  organization  prepares  a  Work  Unit  Summary  document  (table  4.8) 
and  submits  it  electronically  to  DTIC  where  it  is  entered  into  the  appropriate  database. 

A  hypothesis  of  the  proposed  review  process  is  that  the  reviewers,  based  upon  detailed  expert 
knowledge  of  the  particular  CESTs  of  the  work  unit,  can  make  rapid  and  accurate  assessments  of 
the  maturity,  impact  and  feasibility  of  the  work  unit.  The  generation  of  the  CESTs  by  the 
principal  investigator  is  therefore,  the  critical  element  of  the  Work  Unit  Summary.  If  the  CESTs 
are  properly  developed  at  the  level  that  the  work  is  actually  perfonned,  search  engine  technology 
should  be  able  to  retrieve  a  community  of  experts  from  the  published  literature  in  that  field  of 
knowledge.  The  PI  will  also  suggest  experts  that  will  augment  the  automated  search.  Finally,  as  a 
test,  the  PI  will  run  a  search  of  the  CESTs  to  determine  if  an  appropriate  community  of  experts 
(including  some  of  the  suggested  experts)  is  retrieved  by  the  search  engine.  If  not,  the  CEST 
description  must  be  modified  to  produce  the  desired  results. 

2.  The  DTIC  automated  system  extracts  the  CESTs  and  conducts  an  automated  literature  search 
to  identify  a  potential  community  of  experts  for  each  CEST. 

DTIC  is  designated  as  the  custodian  of  the  S&T  Work  Unit  Summary  database.  DTIC  will 
receive  the  data  sheets  from  the  investigating  agency  and  electronically  infonn  DDR&E  that  it 
has  received  the  data.  DTIC  will  then  use  search  engine  technology  to  produce  a  potential 
community  of  experts.  While  this  will  duplicate  to  some  extent  the  search  done  by  the  PI  it  will 
not  be  time  intensive  and  should  be  free  from  any  conflict  of  interest. 

3.  The  suggested  experts  listed  in  the  Work  Unit  Summary  select  the  most  qualified  members 
from  the  DTIC  generated  potential  community  of  experts. 

The  list  of  the  potential  community  of  experts  is  automatically  made  available,  via  a  password 
controlled  web  site,  to  the  suggested  experts  listed  in  the  Work  Unit  Summary.  The  responsible 
organization  will  have  certified  at  the  time  of  submission  of  the  Work  Unit  Summary  document 
that  the  suggested  experts  are  U.S.  citizens  and  have  agreed  to  participate  in  an  assessment.  The 
suggested  experts  place  a  check  mark  next  to  those  individuals  on  the  list  who  they  consider 
appropriate  to  act  as  assessors  for  the  CESTs  in  question.  This  input  combined  with  the 
suggested  experts  will  constitute  the  communities  of  experts  for  the  CESTs. 
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4.  The  selected  expert  review  members  complete  the  appropriate  worksheets  and  submit  them  to 
DTIC. 


The  selected  communities  of  experts  will  be  provided  electronically  a  survey  worksheet  (see 
table  4.9)  to  complete  and  return  to  DTIC.  It  is  expected  that  the  expert  reviewers  will  be  able  to 
perform  the  assessment  in  a  few  minutes  using  their  previous  experience,  the  compact 
assessment  worksheet,  and  the  limited  information  provided  to  them. _ 


Table  4.9:  S&T  Assessment  Worksheet  Assessor: 
Background  Information 

1. Work  Unit  Title  (include  classification  of  title): 

2.  Responsible  Organization: 

3. Critical  enabling  S&T  (include  classification  of 
entries): 

a. Approach 

b.  Metrics 

c.  Milestones 

d.  Funding 

e. Technical  work  years 


Assessment: 

On  a  scale  of  1-10  (1  is  low,  10  is  high)  score  the 
following: 


Topic 

Score 

Comment 

Adequacy  of  funding 

CEST  Properly  Stated 

Realism  of  timeframes 

Level  of  S&T  Challenge 

Impact  if  milestones  are  met 

Level  of  maturity  of  the  S&T 
being  pursued 

Likelihood  of  achieving 

milestones  (item  3.c) 

5.  The  DTIC  automated  system  will  collect  and  analyze  the  input  from  the  various  communities 
of  experts,  identify  areas  for  in  depth  review,  and  prepare  summary  reports  for  the  appropriate 
DDR&E  staff  and  others  (for  example  the  service  S&T  executives  and  the  organizations 
responsible  for  the  work  being  assessed). 


If  the  on-line  review  indicates  the  need  for  an  in  depth  review,  the  same  experts  designated  for 
the  on-line  assessment  are  appropriate  members  for  an  in-depth  review.  An  additional  spin-off  of 
the  process  described  here  for  generating  a  community  of  experts  is  that  this  process  should  be  of 
value  for  selecting  reviewers  for  a  conventional  review  at  the  detailed  technology  level.  While  a 
search  engine  would  pull  up  too  many  names  for  a  broader  subject  area  review,  it  could  still  be 
of  value  in  selecting  subject  experts  to  augment  the  generalists  who  are  typical  of  the 
conventional  review  process. 
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A  simplified  flow  chart  of  the  envisioned  process  is  presented  in  figure  4.2: 


Figure  4.2  EWA  Process  Flow  Chart 
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STEP  3 

The  responsible  organization 
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STEPS 

DTIC  automated  system 
collects  and  analyzes  the 
Results,  identifies  areas  for 
further  review,  and  prepares 
Summary  reports  for  DDR&E 
and  others 


The  state  of  infonnation  technology  today  is  not  advanced  to  the  point  where  it  would  allow  full 
automation  of  the  process  outlined  above.  It  is,  however,  surprisingly  close  to  allowing  such  an 
EWA  to  be  undertaken.  There  are  some  minor  issues  that  must  be  resolved  such  as  those 
associated  with  extracting  individual  authors  and  their  affiliations  from  the  archived  publications 
and  finding  their  e-mail  addresses.  More  serious  issues  must  also  be  resolved  such  as  increasing 
the  search  engine  capabilities  beyond  key  word  searches  to  the  point  where  they  can  reliably 
perform  context  searches.  In  this  regard  the  DOD  should  participate  in  those  efforts  that  are 
ongoing  to  resolve  such  issues.  Active  DTIC  participation  in  the  NIST  and  DARPA  sponsored 
Text  REtrieval  Conference  series  (TREC)  (http://trec.nist.gov/)  is  recommended  for  the  purpose 
of  benchmarking  various  approaches  to  an  EWA.  In  the  short  term  the  techniques  outlined  above 
could  be  profitably  applied  to  the  conventional  review  process  where  manual  intervention  would 
be  substituted  for  those  processes  that  are  not  easily  automated  with  current  technology.  This 
would  allow  the  refining  of  the  data  documentation  requirements. 
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Displaying  Assessment  Results 


The  assessment  methodology  outlined  above  will  generate  a  large  amount  of  data.  One  must, 
therefore,  develop  schemes  for  analyzing  the  data  and  displaying  it  in  ways  that  provide  insights 
into  the  program.  The  last  three  topics  in  table  4.9  are  especially  interesting  in  this  regard.  They 
form  axes  of  the  three-dimensional  data  cube  shown  in  figure  4.3.  The  data  could  be  displayed  as 
three-dimensional  scatter  plots  ranging  from  that  associated  with  a  particular  CEST,  to  a 
combined  scatter  plot  for  all  CESTs  within  a  given  work  unit,  all  the  way  up  to  a  scatter  plot  for 
all  CESTs  in  the  entire  DOD  S&T  program.  These  scatter  plots  would  provide  a  quick  sense  of 
where  in  the  data  cube  (likelihood,  impact,  maturity)  the  DOD  S&T  program  resides  as  a 
function  of  the  level  at  which  it  is  examined. 


S&T  Assessment 


10 


Figure  4.3  Data  Cube 

Three-dimensional  scatter  plots,  while  they  contain  all  of  the  data  are  difficult  to 
interpret/visualize.  It  is  often  useful  to  project  the  data  on  to  the  planes  that  form  the  data  cube. 
An  especially  interesting  display  is  the  projection  of  the  three-dimensional  scatter  plot  onto  the 
Impact-Maturity  plane.  This  plane  could  be  broken  into  quadrants  as  shown  in  figure  4.4.  It 
would  be  quite  illuminating  to  see  where  in  this  plane  the  program  would  cluster.  Clustering  in 
the  lower  left-hand  quadrant  or  the  lower  right-hand  quadrant  would  likely  be  viewed  as 
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undesirable.  Clustering  in  the  upper  left-hand  or  upper  right-hand  quadrants  are  more  interesting 
cases.  There  will  be  very  different  opinions  regarding  in  which  of  these  two  quadrants  the 
program  should  reside.  The  discussion  that  this  data  would  precipitate  would  be  quite  valuable. 
The  projection  of  the  three-dimensional  data  onto  the  other  two  planes  would  also  provide 
valuable  insights  regarding  the  sense  of  the  scientific  and  technical  community  relative  to  the 
program’s  health  and  viability. 


S&T  Assessment 


Figure  4.4  Impact-Maturity  Plane  Template 


The  survey  results  could  be  used  as  a  basis  for  requesting  in-depth  reviews  of  particular  areas. 
For  example,  programs  that  fell  into  the  lower  two  quadrants  of  figure  4.4  could  be  automatically 
identified  for  further  examination.  It  would  also  be  useful  to  correlate  the  Maturity  scores  with 
the  “color”  (6.1,  6.2,  and  6.3)  of  the  money  used  to  fund  the  program.  A  similar  correlation 
should  be  done  of  the  maturity  levels  assigned  by  the  principal  investigators  and  the  maturity 
levels  assigned  by  the  assessors.  In  this  regard,  it  would  be  important  that  the  assessors  are  not 
privy  to  the  maturity  levels  assigned  by  the  principal  investigators.  A  low  degree  of  correlation  in 
either  of  the  above  could  form  the  basis  for  requesting  additional  review. 


In  order  to  provide  a  concrete  example  of  the  data  displays  suggested  above,  the  study  conducted 
a  simulated  assessment  of  the  hypersonic  program  summarized  in  Appendix  A.  The  approach 
used  was  to  establish  a  reasonable  estimate  of  the  mean  scores  (Impact,  Maturity  and  Likelihood) 
that  the  study  directors  would  expect  for  each  of  the  CESTs  if  a  detailed  assessment  were  made. 
The  simulation  then  assumed  50  assessors  whose  scores  were  distributed  normally  along  each 
axis  (Impact,  Maturity,  Likelihood)  about  the  means  with  a  standard  deviation  of  1  (on  a  scale  of 
1-10).  A  small  random  component  was  then  added  to  the  various  scores  to  avoid  the  artifact  of 
all  points  clustering  along  a  line  in  each  two  dimensional  projection.  Figure  4.5  shows  the 
resultant  scatter  plots  for  the  CEST  “Hypersonic  engine  seals”. 
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4.5a  Impact/  Maturity 
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4.5b  Likelihood  /  Maturity 
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Figure  4.5  Scatter  plots  for  the  simulated  assessment  of  CEST  “Hypersonic  engine  seals” 

From  the  above  figures  one  sees  that  the  simulated  assessment  places  this  CEST  in  the  upper 
left-hand  quadrant  of  the  Impact-Maturity  and  Likelihood-Maturity  planes  and  in  the  upper  right- 
hand  quadrant  of  the  Impact-Likelihood  plane.  The  authors  would  view  this  as  satisfactory  for  a 
mature  6.2  or  early  6.3  task. 
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Figure  4.6  present  the  scatter  plots  for  the  CEST  “Thermal  Management  System”. 
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4.6b  Likelihood  /  Maturity 
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4.6c  Impact  /  Likelihood 
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Figure  4.6  Scatter  plots  for  the  simulated  assessment  for  CEST  “Thermal  Management” 


These  scatter  plots  place  the  program  in  the  upper  left  quadrants  of  the  Impact-Maturity  and 
Impact-Likelihood  planes  and  in  the  lower  left  quadrant  of  the  Likelihood-Maturity  plane.  The 
authors  would  interpret  these  results  as  showing  a  satisfactory  6.2  -  6.3  task  but  one  with 
considerably  more  risk  than  the  “hypersonic  seals”  task.  The  risk  is  associated  with  the  technical 
challenges  confronting  this  CEST.  It  is  important  to  note  that  reaching  this  conclusion  requires 
reviewing  all  the  scores  that  are  required  by  table  4.9  and  not  just  the  scores  displayed  in  the 
scatter  plots  shown  above. 


The  scatter  plots  (not  shown)  for  the  simulated  assessment  of  the  “dual  flow-path”  CEST  place 
that  CEST  in  the  upper  right-hand  quadrant  for  all  three  2D  projections  of  the  3D  data  cube. 
Such  an  assessment  suggests  that  the  program  involves  technology  that  has  high  impact,  is 
relatively  mature,  and  is  relatively  low  risk.  One  would  expect  such  a  program  to  be  funded 
primarily  with  6.3  funds. 


The  scatter  plots  (not  shown)  for  the  simulated  assessment  of  the  “control  systems”  CEST  place 
the  CEST  in  the  upper  left-hand  quadrants  for  the  Impact-Maturity  projection  and  the  Impact- 
Likelihood  projection,  but  in  the  lower  left  quadrant  for  the  Likelihood-Maturity  projection.  This 
implies  that  the  program  involves  considerable  risk.  The  high-risk  assessment  in  this  case 
resulted  not  from  the  technical  challenge  confronting  this  CEST  but  rather  from  a  concern  that 
the  funding  provided  was  inadequate  for  the  stated  plans  and  milestones.  Here  again  it  should  be 
noted  that  reaching  this  conclusion  involves  scores  not  shown  by  the  data  cube. 
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The  2D  scatter  plots  that  result  from  combining  the  simulated  assessments  for  all  four  CESTs  are 
shown  in  figure  4.7. 


Maturity 


4.7b  Likelihood  /  Maturity 
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4.7c  Impact  /  Likelihood 


Figure  4.7  Scatter  plots  for  combined  scores  from  the  simulated  assessment  of  the  Work 
Unit  “Hypersonic  Turbine-Based  Combined  Cycle  Engine  (Mach  4  turbine  and  Mach  3.5 
to  7  scramjet  engine)” 


This  figure  suggests  that  the  program  that  is  summarized  in  Appendix  A  is  one  of  high  impact 
and  moderate  maturity  and  one  that  involves  some  considerable  risk.  As  pointed  out  above, 
understanding  this  risk  requires  access  to  data  collected  as  part  of  the  assessment  but  not  shown 
in  figure  4.7.  Since  the  viability  of  the  EWA  approach  suggested  here  requires  that  the  entire 
process  be  automated  it  will  be  necessary  that  simple  algorithms  be  developed  to  flag  concerns. 
For  example,  a  program  that  is  deemed  risky  because  of  the  level  of  the  technical  challenges 
confronting  it  may  very  well  be  a  satisfactory  program.  On  the  other  hand,  a  program  that  is 
deemed  to  be  risky  because  of  inadequate  funding  would  likely  not  be  viewed  as  satisfactory. 
Similarly,  a  program  that  is  assessed  to  be  low  impact  and  low  maturity  might  be  a  satisfactory 
program  because  it  involves  such  new  S&T  that  it  is  difficult  to  judge  its  impact.  History  has 
demonstrated  many  times  that  new  S&T  that  is  originally  deemed  to  be  not  relevant  has  turned 
out  to  be  very  important.  On  the  other  hand,  a  program  that  is  assessed  to  be  low  impact  and  high 
maturity  should  probably  not  be  part  of  the  DOD  S&T  program.  The  development  of  algorithms 
to  sort  out  such  issues  would  be  required  as  part  of  an  EWA.  These  algorithms  could  be  based 
upon  the  type  of  reasoning  given  above.  Figure  4.8  provides  a  simple  decision  template  based 
upon  the  above  considerations. 
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4.8c  Likelihood  /  Maturity 
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Figure  4.8.  Decision  criteria  for  assessments  at  the  individual  work  unit 
level  or  the  individual  CEST  level 

The  above  discussion  covers  but  a  few  of  the  ways  that  the  data  from  an  EWA  could  be 
displayed  and  analyzed.  For  example,  there  are  ten  data  cubes  that  could  be  created  (such  as  a 
cube  with  the  axes:  Adequacy  of  funding,  Realism  of  time  frames,  and  Level  of  scientific  and 
technical  challenge).  Decision  criteria  could  be  established  for  each  of  these  data  cubes.  Also, 
evaluation  of  the  first  few  moments  of  the  rating  distributions  would  be  necessary.  The  matter  of 
how  to  aggregate  the  data  from  the  individual  CEST  scores  is  not  trivial,  especially  as  one  moves 
from  a  few  CESTs  (like  the  case  considered  above)  to  large  numbers  of  CESTs  (such  as  for  the 
total  program).  These  issues  would  best  be  sorted  out  by  conducting  limited  testing  of  the  ideas 
presented  above.  A  reasonable  place  to  start  would  be  to  incorporate  some  of  the  EWA 
considerations  into  the  conventional  review  process  and/or  to  conduct  limited  tests  of  the  EWA 
approach  on  small  segments  of  the  program.  Such  testing  would  provide  guidance  as  to  the 
practicality  and  value  of  the  EWA  that  emerged  from  this  study.  It  would  also  be  an  important 
step  in  gaining  acceptance  of  the  approach  if  it  turned  out  be  practical  and  valuable. 
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5.  Conclusions  and  Recommendations 


Four  major  conclusions  have  emerged  from  this  study: 

•  Many  different  and  valuable  reviews  of  the  DOD  S&T  program  are  currently  conducted. 
However,  the  aggregation  of  these  reviews  does  not  constitute  an  EWA.  There  is  no 
obvious  way  to  modify  the  current  review  structure  so  as  to  produce  an  EWA  as  it  has 
been  defined  for  this  study.  The  underlying  problem  is  the  sheer  magnitude  and  breadth 
of  the  program.  A  new  methodology  will  be  required  if  one  is  to  institutionalize  an  EWA. 

•  The  current  documentation  of  the  DOD  S&T  program  does  not  reflect  the  state  of  the  art 
of  information  technology,  is  marginal  for  exercising  stewardship  over  the  program,  and 
is  not  adequate  for  the  conduct  of  an  EWA.  The  recent  introduction  of  the  DDR&E 
“R&E  Portal”  will  improve  this  situation.  However,  the  planned  documentation  of  the 
program  will  remain  inadequate  for  the  conduct  of  an  EWA. 

•  DOD  does  not  access  systematically  the  breadth  of  scientific  and  technical  knowledge 
and  expertise  that  is  potentially  available  to  it.  The  establishment  of  and  ready  access  to 
communities  of  experts  that  cover  all  DOD  areas  of  scientific  and  technical  interest 
would  be  of  great  value. 

•  The  current  state  of  the  art  in  and  expected  advancements  in  information  technology  and 
library  science  offer  the  best  hope  of  resolving  the  above  concerns. 

These  four  conclusions  lead  to  several  recommendations: 

•  The  principal  investigators  of  the  various  DOD  S&T  projects  should  be  taught  to  and 
required  to  document  their  projects  in  tenns  of  the  very  specific  CEST  issues  that  they 
are  funded  to  resolve.  These  issues  should  be  stated  at  the  level  at  which  people  are 
actually  performing  work  rather  than  in  terms  of  general  objectives. 

•  S&T  project  documentation  requirements  should  be  standardized  across  DOD  and  should 
contain  the  information  needed  for  an  EWA.  Funding  for  a  project  should  not  be  released 
until  the  organization  responsible  for  the  project  submits  satisfactory  documentation  to 
DTIC.  The  DTIC  database  that  results  from  proper  documentation  of  the  program  will  be 
broadly  valuable  across  DOD  (DDR&E,  funding  agencies,  performers,  etc.)  independent 
of  the  EWA.  The  needed  information  is  available  at  the  performer  level  and  can  and 
should  be  documented  as  a  routine  matter.  This  documentation  is  required  as  a  matter  of 
stewardship. 

•  The  underlying  strategy  for  an  EWA  should  be  to  carefully  and  succinctly  state  the  CEST 
issues  being  addressed  by  the  funded  program  so  that  an  expert  in  the  specific  topical 
area  being  pursued  can  quickly  assess  the  viability  of  the  program’s  meeting  stated  plans 
and  milestones. 
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•  National  communities  of  experts  should  be  established  for  each  of  the  specific  CEST 
issues  being  addressed.  The  tools  of  modern  infonnation  technology  and  library  science 
must  be  utilized  to  accomplish  this.  Some  DOD  research  investment  in  library  science 
may  be  required  to  resolve  DOD  specific  issues.  The  establishment  of  the  communities  of 
experts  will  be  of  great  value  independent  of  the  EWA. 

•  For  an  EWA,  the  approach  of  collecting  the  review  teams  at  a  central  location  for  the 
purpose  of  holding  hearings  must  be  abandoned  and  replaced  by  a  distributed  assessment 
process  that  exploits  the  documentation  of  all  of  the  CEST  issues,  the  advances  in 
information  technology  and  library  science,  and  the  readily  available  access  to 
geographically  distributed  communities  of  experts.  The  EWA  should  be  conducted  as  a 
targeted  survey  of  the  appropriate  communities  of  experts. 

•  DTIC  should  be  the  custodian  of  the  new  DOD  S&T  database  and  for  the  automation  of 
and  the  routine  implementation  of  the  EWA.  The  assessment  should  ultimately  be  a 
routine  byproduct  of  doing  business. 

•  A  full  EWA  as  envisioned  by  this  study  should  not  be  initiated  in  the  short  tenn.  Many 
details  need  to  be  addressed  before  such  an  assessment  could  be  undertaken.  Certain 
aspects  of  the  proposed  process  should  be  examined  via  the  conventional  review  process. 
For  example,  automated  techniques  for  forming  communities  of  experts  could  be  tested 
by  employing  them  as  part  of  establishing  the  conventional  review  teams.  Test  cases 
should  be  conducted  to  clarify  how  to  document  the  program  in  terms  of  CEST  and  to 
examine  and  resolve  problems  that  confront  the  distributed  assessment  approach.  These 
test  cases  should  address  classified  distributed  assessments  (e.g.  utilizing  SIPRnet)  and 
unclassified  distributed  assessments.  DOD  should  be  an  active  participant  in  the  national 
efforts  that  are  underway  to  take  advantage  of  the  ongoing  revolution  in  library  science. 


If  the  process  described  in  this  report  could  be  implemented  it  would  have  a  number  of  spin-offs. 
For  example,  the  supporting  DTIC  database  could  be  interrogated  so  as  to  compare  the  actual 
program  of  record  with  the  DOD  S&T  investment  strategy.  It  would  also  establish  a  broad 
database  of  experts  covering  most  of  the  S&T  areas  of  interest  to  DOD.  It  may  be  that  the  most 
valuable  result  of  this  approach  to  an  EWA  is  the  establishment  and  maintenance  of  a  detailed 
database  for  the  funded  S&T  program  and  the  development  of  a  large  community  of  experts  who 
have  a  connection  with  and  interest  in  DOD  S&T. 
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APPENDIX  A 


Work  Unit  Summary  Hypothetical  Example 

1.  Title:  Hypersonic  Turbine-Based  Combined  Cycle  Engine  (Mach  4  turbine  and  Mach  3.5  to  7 
scramjet  engine)  (U) 


2.  Date  of  Summary:  12/08/05 


3.  Responsible  Organization 

a.  Name:  Some  DOD  Activity  b.  Address:  Somewhere,  USA 


c.  Principle  Investigator: 


Name 

Phone 

Email  Address 

John  Doe 

123-456-7891 

jdoe@ — .mil 

4.  Classification  Level: 

a:  Summary  Level:  U  b:  Work  Level:  U 

5.  Military  need:  Technology  relevant  to  candidate  system  solutions  for  Prompt  Global  Strike,  Long 
Range  Strike,  and  Assured  Access  to  Space: 

•  Responsive  global  power  projection 

•  Responsive  and  persistent  ISR 

•  Responsive  global  payload  delivery 

•  Responsive  space  payload  delivery 


6.  Technical  Objective:  Future  hypersonic  air-breathing  UCAVs  for  time-critical  theater  operations,  and 
reusable  launch  vehicles  for  prompt  global  response  and  routine  space  access,  will  require  2-3 
propulsion  modes  to  operate  across  their  design  speed  ranges.  One  of  the  most  likely  propulsion 
systems  for  such  applications  will  be  the  turbine-based  combined-cycle  engine  (TBCC),  which  consists 
of  a  supersonic  (-Mach  4)  turbine  engine  closely  integrated  with  a  dual-mode  ramjet/scramjet  engine. 
In  order  to  proceed  along  this  path  it  will  be  necessary  to  develop  a  turbine-based  combined-cycle 
(TBCC)  engine  inlet  system  that  efficiently  delivers  air  flow  required  by  the  turbine  engine  when 
operating  alone,  air  flow  required  by  the  scramjet  when  operating  alone,  and  to  both  engines 
simultaneously  during  the  transition  from  turbine  to  scramjet.  It  is  also  necessary  to  maintain  inlet  fluid 
dynamic  stability  during  mode  transition  and  in  all  other  operational  engine  modes.  One  must  develop 
a  TBCC  engine  nozzle  system  that  efficiently  expands  flow  to  produce  thrust  for  the  turbine  engine 
when  operating  alone,  for  the  scramjet  engine  when  operating  alone,  and  for  both  engines  when 
operating  simultaneously  during  mode  transition.  It  is  necessary  to  develop  an  aircraft  and  engine 
flow-path  control  system  that  adequately  stabilizes  and  controls  the  aircraft  and  TBCC  engine  flow- 
paths  during  engine  mode  transition  and  all  other  flight  phases.  Lastly,  it  will  be  necessary  to  develop 
the  material,  structural  and  mechanical  design  for  the  articulated  TBCC  inlet  and  nozzle  systems,  and  a 
thennal  management  system  for  the  turbine  engine  when  it  is  shutdown  but  exposed  to  scramjet  flow- 
path  heating 
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7.  Progress:  New  project 


8.  Critical  Enabling  Science  and  Technology  (CEST) 


CEST  1:  Hypersonic  engine  dynamic  seals  for  articulated  (i.e.  variable  geometry)  inlet  and  nozzle 
structural  elements.  (U) 

Approach:  Design  and  test  high-temperature  seals  for  a  full-scale  TBCC  inlet  and  nozzle  (Mach  4 
turbine  and  Mach  3.5  to  7  scramjet  engine).  Consideration  will  be  given  to  both  metallic  and  ceramic 
seals. 

Technology  Metrics: 


Metric 

Metric  Unit 

SOA* 

Planned  Value 

Maximum  use 
temperature 

Temperature  °  F 

1400°  F  (metallic) 
1800°  F  (ceramic) 

1800°  F  (metallic) 

2500-3000°  F  (ceramic) 

Durability/life 

#  of  cycles  before 
replacement 

1000’s  for  metallic 
10-  100  for 
ceramic 

1 000’ s  for  <2000°  F 

100’s  at  2500°  F 

10’s  at  3000°  F 

Milestones: 


TRL 

Year 

Description 

6 

2008 

Design  and  fabricate  seals  and 
complete  critical  component  testing  in 
a  relevant  environment 

*Current  TRL  =  2-3 


Funding: 


Fiscal  Year 

2005 

2006 

2007 

2008 

Funding 

Profile 

$1200K 

$1500K 

$1800K 

$1500K 

Technical  Work  Years 


Fiscal  Year 

2005 

2006 

2007 

2008 

Technical 

Work  Years 

3 

4 

4 

3 

State  of  the  art. 
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Suggested  Community  of  Experts: 


Last  Name* 

First  Name* 

Organization 

Email 

Phone 

NASA  Glenn 
Research  Center 

NASA  Glenn 
Research  Center 

*names  deleted 

1 

CEST  2:  Dual  flow-path  TBCC  hypersonic  engine  inlets  and  nozzles  (U) 

Approach:  Define  TBCC  (Mach  4  turbine  and  Mach  3.5  to  7  scramjet  engine)  inlet  and  nozzle 
geometry  and  articulation  requirements  based  initially  upon  knowledge  of  existing  supersonic  inlets 
and  nozzles,  empirical  and  theoretical  design  guidelines  and  operability  (i.e.,  stability)  criteria,  engine 
performance  characteristics,  and  analysis  results  derived  from  engineering  codes  and  Computational 
Fluid  Dynamics  (CFD).  This  would  be  followed  by  building  sub-scale  inlet  and  nozzle  models  and 
testing  them  in  wind  tunnels,  both  statically  and  with  dynamic  articulation  of  the  variable  geometry 
components.  The  knowledge  gained  from  testing  and  analysis  will  be  used  to  optimize  inlet  and  nozzle 
designs.  This  will  be  aided  by  mathematical  optimization  and  multidisciplinary  design  optimization 
techniques. 


Technology  Metrics: 


Metric 

Metric  Unit 

SOA 

Planned  Value 

Integrated  dual  inlet 
operability 

#  of  inlets  operating 
simultaneously  at  high 
performance 

1 

2  at  SOA  inlet  recovery 
and  operability/stability 

Integrated  dual  nozzle 
operability 

#  of  nozzle  operating 
simultaneously  at  high 
performance 

1 

2  at  SOA  thrust 
coefficient 

Milestones: 


TRL 

Year 

Description 

3 

2006 

TBCC  inlet  and  nozzle  design  and 
analysis 

4 

2007 

Wind  tunnel  test  sub-scale  TBCC 
inlet  and  nozzle 

5 

2008 

Update  inlet  and  nozzle  designs,  and 
tunnel  test  inlet  on  a  sub-scale  aircraft 
model  with  an  articulated  inlet 

*Current  TRL  =  2-3 
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Funding: 


Fiscal  Year 

2005 

2006 

2007 

2008 

Funding  Profile 

0 

$3000K 

$4000K 

$3000K 

Technical  Work  Years 


Fiscal  Year 

2005 

2006 

2007 

2008 

Technical 

Work  Years 

0 

8 

8 

8 

Suggested  Community  of  Experts: 


Last  Name* 

First  Name* 

Organization 

Email 

Phone 

Johns  Hopkins 

Applied  Physics 
Laboratory 

Air  Force  Research 

Laboratory/Propulsion 

Directorate 

NASA  Langley 
(Retired) 

TechLand  Research, 
Inc. 

NASA  Langley 
(Retired) 

*names  deleted 


CEST  3:  Post  shutdown  turbine  hypersonic  engine  thermal  management  system  (TMS)  (U) 

Approach:  Design  a  thennal  management  system  (TMS)  for  a  TBCC  engine  (Mach  4  turbine  and 
Mach  3.5  to  7  scramjet  engine)  flow-path  and  analyze  it  using  fluid,  thermal  and  structural  analysis 
codes,  and  then  bench  test  critical  TMS  components  to  verify  predicted  operation  and  perfonnance. 


Technology  Metrics: 


Metric 

Metric  Unit 

SOA 

Planned  Value 

Turbine  bay 
temperature  during 
scramjet  operation 

Temperature 

N/A 

Max  temperature 
capability  of  lowest 
temperature  turbine 
material  (~  1500°  F) 
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Milestones: 


TRL 

Year 

Description 

3 

2007 

TMS  design  and  analysis 

4 

2008 

Critical  TMS  component 
bench  testing  and  analysis 
update 

Funding: 


Fiscal  Year 

2005 

2006 

2007 

2008 

Funding  Profile 

NA 

$1000k 

$2000k 

$3000k 

Technical  Work  Years 


Fiscal  Year 

2005 

2006 

2007 

2008 

Technical 

Work  Years 

0 

3 

4 

5 

Suggested  Community  of  Experts: 


Fast  Name* 

First  Name* 

Organization 

Email 

Phone 

Pratt  and  Whitney 

Pratt  and  Whitney 

Pratt  and  Whitney 

Air  Force  Research  Fab 

Pratt  and  Whitney,  retired 

*names  deleted 

CEST  4:  Integrated  high  speed  airframe  and  engine  control  system  development  (U) 

Approach:  Develop  inlet  and  nozzle  control  algorithms  for  a  hypersonic  Turbine-Based  Combined 
Cycle  Engine  (Mach  4  turbine  and  Mach  3.5  to  7  scramjet  engine),  and  then  test  them  via  dynamic 
simulation  using  data  derived  from  testing  and  CFD  analysis.  Vehicle  stability  and  control  during 
engine  mode  transition  will  be  verified  via  dynamic  simulation  using  CFD-derived  aero  data  in 
conjunction  with  inlet  and  nozzle  test  data.  Wind  tunnel  tests  will  be  conducted  of  an  aircraft  model 
with  variable  inlet  to  verily  inlet  stability  during  mode  transition,  and  aero  data  will  be  gathered  to 
verify  vehicle  stability  and  control  during  the  transition  event. 
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Technology  Metrics 


Metric 

Metric  Unit 

SOA 

Planned  Value 

Inlet  stability  during 
engine  mode  transition 

Stability  margin 

N/A 

5% 

Aerodynamic  dynamic 
stability  during  engine 
mode  transition 

Control  system  phase 
and  gain 

N/A 

6  db  gain  and  45° 
phase  margin 

Milestones 


TRL 

Year 

Description 

3 

2007 

Develop  inlet,  nozzle  and  aircraft 
control  laws  using  aerodynamic 
propulsion  analysis  of  same. 
Perform  dynamic  simulations. 

4 

2008 

Use  inlet  control  laws  in  wind 
tunnel  test  of  integrated 
inlet/airframe.  Use  tunnel  data  to 
update  control  laws  and 
simulations 

*Current  TRL  =  2-3 


Funding 


Fiscal  Year 

2005 

2006 

2007 

2008 

Funding 

Profile 

$1000  k 

$1500  k 

$1500  k 

$2000  k 

Technical  Work  Years 


Fiscal  Year 

2005 

2006 

2007 

2008 

Technical 

Work  Years 

3 

3 

3 

4 

Suggested  Community  of  Experts: 


Last  Name* 

First  Name* 

Organization 

Email 

Phone 

Boeing  Huntington  Beach 

NASA  Dryden 

Air  Force  Research  Laboratory 

Boeing  St.  Louis 

*names  deleted 
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9.  Total  Project  Funding: 


Fiscal  Year 

2005 

2006 

2007 

2008 

Total 

Funding 

$2200k 

$7000k 

$9300k 

$9500k 

10.  Program  Participants: 


Contracts  with  whom 

XYZ  Corp. 

Grants  with  whom 

TBD 

Government  organization 

Some  DOD  Activity 

11.  DOD  Subject  Areas:  (http://www.dtic.mil/dtic/subcatguide/  ) 

•  01-01-Aerodynamics 

•  01-03-Aircraft 

•  01-04-Flight  control  and  instrumentation 

•  2 1-05- Jet  and  gas  turbine  engines 

12.  Key  Words:  Hypersonic  turbine-based  combined-cycle  engine;  High-temperature  seals; 
Supersonic  inlet;  Supersonic  nozzle;  Turbine  engine;  Scramjet  engine;  Supersonic  inlet  control  system; 
Variable  geometry  inlet;  Variable  geometry  nozzle. 
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APPENDIX  B 

Potential  Communities  of  Experts 

The  potential  community  of  experts  shown  here  were  obtained  by  on-line  computer  searches  of  various 
databases  using  search  engines  such  as  Scirus,  Google,  Google  Scholar,  and  the  A1AA  Electronic  Library 

Table  Bl:  Potential  community  of  experts  for  CEST  1:  Hypersonic  Engine  Seals 


Number  of  Individuals 
Identified 

Organization 

Organization  Location 

1 

Advanced  Components  &  Materials,  Inc 

East  Greenwich,  Rl 

2 

Aerojet 

Sacramento,  CA 

2 

Aerothermal  and  Aero-Optics  Evaluation 
Center 

Buffalo,  NY 

1 

Albany  Techniweave,  Inc. 

Albany,  NY 

8 

ATKThiokol  Propulsion  Corporation 

Brigham  City,  UT 

2 

BF  Goodrich  Aerospace 

Chula  Vista,  CA 

1 

California  Institute  of  Technology 

Pasadena,  CA 

4 

Case  Western  Reserve  University 

Cleveland,  OH 

1 

Connecticut  Reserve  Technologies 

Stow,  OH 

3 

Drexel  University 

Philadelphia,  PA 

1 

Embry-Riddle  Aeronautical  University 

Daytona  Beach,  FL 

4 

FDC/NYMA,  Inc. 

Hampton,  VA 

3 

General  Applied  Science  Laboratories,  Inc. 

Ronkonkoma,  NY 

2 

General  Electric 

Cincinnati,  OH 

2 

General  Electric  Global  Research  Center 

Niskayuna,  NY 

1 

JIAFS,  The  George  Washington  University 

Hampton,  VA 

9 

Johns  Hopkins  University,  Applied  Physics 

Laurel,  MD 

1 

Modern  Technologies  Corporation 

Middleburg  Heights,  OH 

3 

Mohawk  Innovative  Technology,  Inc. 

Albany,  NY 

2 

NASA  Ames  Research  Center 

Moffett  Field,  CA 

10 

NASA  Glenn  Research  Center 

Cleveland,  OH 

1 

NASA  Headquarters 

Washington,  DC 

28 

NASA  Langley  Research  Center 

Hampton,  VA 

10 

NASA  Lewis  Research  Center 

Cleveland,  OH 

2 

NASA  Marshall  Space  Flight  Center 

Huntsville,  AL 

2 

Ohio  Aerospace  Institute 

Beavercreek,  OH 

4 

Orbital  Sciences  Corporation 

Dulles,  VA 

2 

Pennsylvania  State  University 

University  Park,  PA 

1 

Refractory  Composites  Inc. 

Glen  Burnie,  MD 

1 

Rensselaer  Polytechnic  Institute 

Troy,  NY 

1 

Saint-Gobain  Advanced  Ceramics 

Northboro,  MA 

2 

Sverdrup  Technology,  Inc. 

Brook  Park,  OH 

1 

The  Advanced  Products  Company 

North  Haven,  CT 

3 

The  Boeing  Company 

Long  Beach,  CA 

1 

Tribos  Engineering,  P.C 

Niskayuna,  NY 

7 

U.S.  Air  Force  Research  Laboratory 

Wright-Patterson  Air  Force  Base,  OH 

1 

U.S.  Army  Research  Lab 

Adelphi,  MD 

1 

Universal  Technology  Corporation 

Beavercreek,  OH 

3 

University  of  Akron 

Akron,  OH 

1 

University  of  Minnesota 

Minneapolis,  MN 

1 

University  of  Toledo 

Toledo,  OH 

1 

Virginia  Tech,  Mechanical  Engineer  Dept. 

Blacksburg,  VA 
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Table  B2:  Potential  community  of  experts  for  CEST  2:  Hypersonic  Turbine-Based  Combined 
Cycle  Engine  (Mach  4  turbine  and  Mach  3.5  to  7  scramjet  engine) 


Number  of  Individuals 
Identified 

Organization 

Organization  Location 

2 

Allison  Advanced  Development  Co. 

Indianapolis,  IN 

2 

Astrox  Corporation 

College  Park,  MD 

3 

Boeing  Corporation 

St.  Louis,  MO 

1 

Brigham  Young  University 

Provo,  UT 

1 

Johns  Hopkins  University 

Columbia,  MD 

1 

Johns  Hopkins  University,  Applied  Physics 

Laurel,  MD 

8 

NASA  Langley  Research  Center 

Hampton,  VA 

1 

NASA  Marshall  Space  Flight  Center 

Huntsville,  AL 

1 

Pyrodyne  Inc. 

Glenwood,  MD 

1 

SAF/AQRT 

Washington,  DC 

2 

Spiritech  Advanced  Products  Inc. 

Jupiter,  FL 

2 

U.S.  Air  Force  Research  Lab 

Wright-Patterson  AFB,  OH 

1 

University  of  Florida 

Gainesville,  FL 

2 

University  of  Maryland 

College  Park,  MD 

Table  B3.  Potential  community  of  experts  for  CEST  3:  High-speed  turbine  engine  thermal 
management  system  (TMS) 


Number  of  Individuals  Identified 

Organization 

Organization  Location 

2 

Aerojet 

Sacramento,  CA 

5 

Air  Force  Research  Laboratory 

Wright  Patterson  AFB,  OH 

2 

Allison  Advanced  Development  Co. 

Indianapolis,  IN 

1 

Andrews  Space  Inc. 

Seattle,  WA 

1 

Brigham  Young  University 

Provo,  UT 

4 

FDC/NYMA,  Inc. 

Hampton,  VA 

1 

Flight  Unlimited 

Flagstaff,  AZ 

4 

Georgia  Institute  of  Technology 

Atlanta,  GA 

1 

Johns  Hopkins  University 

Columbia,  MD 

2 

NASA  Dryden  Flight  Research  Center 

Edwards,  CA 

7 

NASA  Glenn  Research  Center 

Cleveland,  OH 

1 

NASA  Headquarters 

Washington,  DC 

15 

NASA  Langley  Research  Center 

Hampton,  VA 

1 

NASA  Marshall  Space  Flight  Center 

Huntsville,  AL 

1 

Parks  College 

St.  Louis,  MO 

1 

Pratt  &  Whitney  Space  Propulsion 

West  Palm  Beach,  FL 

6 

SAIC 

Huntsville,  AL 

3 

SAIC 

Torrance,  CA 

3 

Space  Works  Engineering,  Inc. 

Atlanta,  GA 

2 

Spiritech  Advanced  Products,  Inc. 

Jupiter,  FL 

3 

The  Boeing  Company 

Long  Beach,  CA 

3 

The  Boeing  Company 

Seal  Beach,  CA 

1 

University  of  Florida 

Gainesville,  FL 
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Table  B4.  Potential  community  of  experts  for  CEST  4:  Hypersonic  Engine  Control  System 
Development 


Number  of  Individuals  Identified 

Oraanization 

Oraanization  Location 

5 

Air  Force  Research  Laboratory 

Wright-Patterson  AFB,  OH 

4 

Andrews  Space,  Inc. 

Seattle,  WA 

1 

Boeing  Company 

St.  Louis,  MO 

1 

Boeing  Phantom  Works 

Huntsville,  AL 

2 

Boeing  Phantom  Works 

St.  Louis,  MO 

1 

Brigham  Young  University 

Provo,  Utah 

2 

Enercon  Systems 

Cleveland,  OH 

3 

FDC/NYMA,  Inc.  Aerospace  Sector 

Hampton,  VA 

1 

Flight  Unlimited 

Flagstaff,  AZ 

1 

Flow  Parametrics 

New  Castle,  DE 

3 

General  Applied  Science 
Laboratories 

Ronkonkoma,  NY 

1 

General  Electric 

Cincinnati,  OH 

1 

General  Electric 

Lynn,  MA 

6 

Georgia  Institute  of  Technology 

Atlanta,  GA 

1 

International  Space  Systems,  Inc. 

Huntsville,  AL 

2 

Johns  Hopkins  University 

Columbia,  MD 

1 

Modern  Technologies  Corporation 

Middleburg  Heights,  OH 

2 

NASA  Ames  Research  Center 

Moffett  Field,  CA 

3 

NASA  Dryden  Flight  Research 
Center 

Edwards,  CA 

9 

NASA  Glenn  Research  Center 

Cleveland,  OH 

1 

NASA  Headquarters 

Washington,  DC 

32 

NASA  Langley  Research  Center 

Hampton,  VA 

6 

NASA  Lewis  Research  Center 

Cleveland,  OH 

1 

NASA  Marshall  Space  Flight  Center 

Huntsville,  AL 

3 

Ohio  State  University 

Columbus,  OH 

1 

Parks  College 

St.  Louis,  MO 

2 

Pennsylvania  State  University 

University  Park,  PA 

3 

Princeton  University 

Princeton,  NJ 

1 

Proton  Aerospace  Corporation 

Jupiter,  FL 

1 

QSS  Group,  Inc. 

Cleveland,  OH 

8 

SAIC 

Huntsville,  AL 

2 

Spiritech  Advanced  Products,  Inc. 

Jupiter,  FL 

2 

TechLand  Research,  Inc. 

North  Olmstead,  OH 

4 

The  Boeing  Company 

Seal  Beach,  CA 

1 

University  of  Alabama 

Huntsville,  AL 

1 

University  of  Florida 

Gainesville,  Florida 

2 

University  of  Maryland 

College  Park,  MD 
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NASA  Technology  Inventory 

This  information  presented  in  this  appendix  is  taken  from  several  sites  that  describe  the  NASA 
Technology  Inventory.  The  purpose  of  the  NASA  Technology  Inventory  is  to  document  the  full  scope 
of  NASA's  technology  investment  in  order  to  improve  management  and  communications  inside  and 
outside  the  agency.  Each  of  the  5  NASA  Enterprises  manages  a  technology  program  that  is  focused  on 
the  needs  and  objectives  of  that  Enterprise.  In  addition,  the  enterprises  support  technology  activities 
within  their  Advanced  Development,  and  Research  and  Analysis  Programs.  Other  NASA  programs 
contribute  technology  as  well,  including  the  Cross  Enterprise  Technology  Development  Program,  the 
Small  Business  Innovation  Research  Program,  the  Center  Director's  Discretionary  Program,  safety  and 
engineering  programs,  and  educational  programs.  Collectively,  all  of  the  technology  activities  in 
NASA  amount  to  about  $1.6  billion. 

The  Tech  Inventory  seeks  to  capture  this  spectrum  of  activities  in  a  common,  searchable  format.  Each 
input  is  linked  to  the  Enterprises  that  would  benefit,  so  that  program  managers  can  quickly  identify 
technologies  that  support  their  programs.  Keywords  and  a  search  engine  enable  technologists  to 
identify  related  work.  Technology  Readiness  Levels  help  describe  the  maturity  of  programs,  and  help 
separate  near-term  from  far-term  work.  Enterprise-specific  fields  help  NASA  managers  track  their 
technology  programs.  Partnership  infonnation  is  collected  as  a  NASA  GPRA  metric  that  helps 
measure  the  extent  of  collaboration  with  other  agencies. 

For  the  purposes  of  this  database,  technology  is  defined  as  the  practical  application  of  knowledge  to 
create  the  capability  to  do  something  entirely  new  or  in  an  entirely  new  way.  This  can  be  contrasted  to 
"scientific  research,"  which  encompasses  the  discovery  of  new  knowledge  from  which  new  technology 
is  derived,  and  engineering,  which  uses  technology  derived  from  this  knowledge  to  solve  specific 
technical  problems.  When  investments  are  made  in  a  particular  technology,  it  begins  to  mature — a 
process  of  testing  and  analysis  that  progressively  reduces  the  programmatic  risk  of  selecting  that 
technology  for  an  application  and  increases  the  readiness  of  that  technology  for  use  in  a  mission. 
Technology  may  be  described  in  terms  of  maturity  within  a  scale  of  Technology  Readiness  Levels, 
which  reflect  the  extent  to  which  the  technology  has  been  proven  in  a  realistic  situation. 

Technology  Readiness  Levels  (TRLs)  are  a  systematic  metric/measurement  system  that  supports 
assessments  of  the  maturity  of  a  particular  technology  and  the  consistent  comparison  of  maturity 
between  different  types  of  technology.  The  TRL  concept  is  based  on  a  general  model  for  technology 
maturation  that  includes:  (a)  research  in  new  technologies  and  concepts  (targeting  identified  goals,  but 
not  necessary  specific  systems),  (b)  technology  development  addressing  specific  technologies  for  one 
or  more  potential  identified  applications,  (c)  technology  development  and  demonstration  for  each 
specific  application  before  the  beginning  of  full  system  development  of  that  application,  (d)  system 
development  (through  first  unit  fabrication),  and  (e)  system  launch  and  operations. 
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Technology  Readiness  Levels  Summary 


TRL  1  Basic  principles  observed  and  reported 

TRL  2  Technology  concept  and/or  application  formulated 

TRL  3  Analytical  and  experimental  critical  function  and/or  characteristic  proof-of-concept 
TRL  4  Component  and/or  breadboard  validation  in  laboratory  environment 
TRL  5  Component  and/or  breadboard  validation  in  relevant  environment 

TRL  6  System/subsystem  model  or  prototype  demonstration  in  a  relevant  environment  (ground  or 
space) 

TRL  7  System  prototype  demonstration  in  a  space  environment 


Discussion  of  Each  Level 


The  following  paragraphs  provide  a  descriptive  discussion  of  each  technology  readiness  level, 
including  an  example  of  the  type  of  activities  that  would  characterize  each  TRL. 

TRL  1 

Basic  principles  observed  and  reported 

This  is  the  lowest  "level"  of  technology  maturation.  At  this  level,  scientific  research  begins  to  be 
translated  into  applied  research  and  development.  Examples  might  include  studies  of  basic  properties 
of  materials  (e.g.,  tensile  strength  as  a  function  of  temperature  for  a  new  fiber). 

TRL  2 

Technology  concept  and/or  application  formulated 

Once  basic  physical  principles  are  observed,  then  at  the  next  level  of  maturation,  practical  applications 
of  those  characteristics  can  be  invented  or  identified.  For  example,  following  the  observation  of 
high  critical  temperature  superconductivity,  potential  applications  of  the  new  material  for  thin  film 
devices  (e.g.,  SIS  mixers)  and  in  instrument  systems  (e.g.,  telescope  sensors)  can  be  defined.  At  this 
level,  the  application  is  still  speculative:  there  is  not  experimental  proof  or  detailed  analysis  to  support 
the  conjecture. 

TRL  3 

Analytical  and  experimental  critical  function  and/or  characteristic  proof-of-concept 

At  this  step  in  the  maturation  process,  active  research  and  development  (R&D)  is  initiated.  This  must 
include  both  analytical  studies  to  set  the  technology  into  an  appropriate  context  and  laboratory-based 

C2 


studies  to  physically  validate  that  the  analytical  predictions  are  correct.  These  studies  and  experiments 
should  constitute  "proof-of-concept"  validation  of  the  applications/concepts  formulated  at  TRL  2.  For 
example,  a  concept  for  High  Energy  Density  Matter  (HEDM)  propulsion  might  depend  on  slush  or 
super-cooled  hydrogen  as  a  propellant:  TRL  3  might  be  attained  when  the  concept-enabling 
phase/temperature/pressure  for  the  fluid  was  achieved  in  a  laboratory. 

TRL  4 

Component  and/or  breadboard  validation  in  laboratory  environment 

Following  successful  "proof-of-concept"  work,  basic  technological  elements  must  be  integrated  to 
establish  that  the  "pieces"  will  work  together  to  achieve  concept-enabling  levels  of  performance  for  a 
component  and/or  breadboard.  This  validation  must  be  devised  to  support  the  concept  that  was 
formulated  earlier,  and  should  also  be  consistent  with  the  requirements  of  potential  system 
applications.  The  validation  is  relatively  "low-fidelity"  compared  to  the  eventual  system:  it  could  be 
composed  of  ad  hoc  discrete  components  in  a  laboratory.  For  example,  a  TRL  4  demonstration  of  a 
new  fuzzy  logic  approach  to  avionics  might  consist  of  testing  the  algorithms  in  a  partially  computer- 
based,  partially  bench-top  component  (e.g.,  fiber  optic  gyros)  demonstration  in  a  controls  lab  using 
simulated  vehicle  inputs. 

TRL  5 

Component  and/or  breadboard  validation  in  relevant  environment 

At  this,  the  fidelity  of  the  component  and/or  breadboard  being  tested  has  to  increase  significantly.  The 
basic  technological  elements  must  be  integrated  with  reasonably  realistic  supporting  elements  so  that 
the  total  applications  (component-level,  sub-system  level  or  system-level)  can  be  tested  in  a  simulated 
or  somewhat  realistic  environment.  From  one-to-several  new  technologies  might  be  involved  in  the 
demonstration.  For  example,  a  new  type  of  solar  photovoltaic  material  promising  higher  efficiencies 
would  at  this  level  be  used  in  an  actual  fabricated  solar  array  blanket  that  would  be  integrated  with 
power  supplies,  supporting  structure,  etc.,  and  tested  in  a  thermal  vacuum  chamber  with  solar 
simulation  capability. 

TRL  6 

System/subsystem  model  or  prototype  demonstration  in  a  relevant  environment  (ground  or  space) 

A  major  step  in  the  level  of  fidelity  of  the  technology  demonstration  follows  the  completion  of  TRL  5. 
At  TRL  6,  a  representative  model  or  prototype  system  or  subsystem  which  would  go  well  beyond  ad 
hoc,  patch-cord  or  discrete  component  level  breadboard  would  be  tested  in  a  relevant  environment.  At 
this  level,  if  the  only  relevant  environment  is  the  environment  of  space,  then  the  model/prototype  must 
be  demonstrated  in  space.  Of  course,  the  demonstration  should  be  successful  to  represent  a  true  TRL  6. 
Not  all  technologies  will  undergo  a  TRL  6  demonstration:  at  this  point  the  maturation  step  is  driven 
more  by  assuring  management  confidence  than  by  R&D  requirements.  The  demonstration  might 
represent  an  actual  system  application,  or  it  might  only  be  similar  to  the  planned  application,  but  using 
the  same  technologies.  At  this  level,  several-to-many  new  technologies  might  be  integrated  into  the 
demonstration.  For  example,  a  innovative  approach  to  high  temperature/low  mass  radiators,  involving 
liquid  droplets  and  composite  materials,  would  be  demonstrated  to  TRL  6  by  actually  flying  a  working, 
sub-scale  (but  scaleable)  model  of  the  system  on  a  Space  Shuttle  or  International  Space  Station  pallet. 
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In  this  example,  the  reason  space  is  the  relevant  environment  is  that  microgravity  plus  vacuum  plus 
thennal  environment  effects  will  dictate  the  success/failure  of  the  system  and  the  only  way  to  validate 
the  technology  is  in  space. 

TRL  7 

System  prototype  demonstration  in  a  space  environment 

TRL  7  is  a  significant  step  beyond  TRL  6,  requiring  an  actual  system  prototype  demonstration  in  a 
space  environment.  It  has  not  always  been  implemented  in  the  past.  In  this  case,  the  prototype  should 
be  near  or  at  the  scale  of  the  planned  operational  system  and  the  demonstration  must  take  place  in 
space.  The  driving  purposes  for  achieving  this  level  of  maturity  are  to  assure  system  engineering  and 
development  management  confidence  (more  than  for  purposes  of  technology  R&D).  Therefore,  the 
demonstration  must  be  of  a  prototype  of  that  application.  Not  all  technologies  in  all  systems  will  go  to 
this  level.  TRL  7  would  normally  only  be  perfonned  in  cases  where  the  technology  and/or  subsystem 
application  is  mission  critical  and  relatively  high-risk.  Example:  the  Mars  Pathfinder  Rover  is  a  TRL  7 
technology  demonstration  for  future  Mars  micro-rovers  based  on  that  system  design.  Example:  X- 
vehicles  are  TRL  7,  as  are  the  demonstration  projects  planned  in  the  New  Millennium  spacecraft 
program. 

The  NASA  Technology  Inventory  is  available  to  government  employees  and  a  restricted  version  is 
available  to  government  contractors.  This  study  accessed  the  inventory  and  found  its  functionality  to 
be  quite  good.  It  was  straightforward  to  extract  infonnation  from  the  inventory  and  reconfigure  it  for 
specific  display  purposes.  The  following  sequence  of  charts  represents  one  such  reconfiguration  and  is 
meant  to  illustrate  how  one  can  move  from  the  highest  level  of  description  in  the  inventory  to  its 
lowest  level.  Each  chart  in  the  sequence  was  arrived  at  by  a  mouse  click  on  a  desired  part  of  the 
previous  chart.  The  first  chart  is  the  highest  level  of  aggregation  and  the  last  chart  is  the  lowest  level. 
The  example  sequence  tracks  the  Technology  Discipline  “Power”.  The  specific  details  of  the  program 
have  been  removed  since  they  are  not  relevant  to  the  purpose  here  where  only  functionality  is  of 
interest. 
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NASA  Technology  Disciplines:  Funding  in  $K 

Total  Funding  =  $3, 951, 991 K 


Training  Technology 
Thermal  Protection  Systems 
Thermal  Control 

Systems  Engineerin 


Advanced  Concept  Analysis 


Structures/Materials 


Space  Environmental  Effect] 
Robotics 


Propulsio 


Aero  &  Aerothermodynamics 


Autonomy 


Avionics 


Biotechnology 


Communications 


Cryogenics 

Education 


Power 

Medicine/Health  & 

Performance  Information  Systems\  EVA 

Life  Support  Manufacturing/Processing 

Instrumentation  Instruments 
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NASA  Technology  Discipline  Power:  Funding 


Total  Funding  =  $60,097K 


Technology  Discipline 


Storage 
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Power:  Generation  Funding  Breakdown  ($K) 

Total  Funding  =$24,162.14K 


□Task  ID: 
■Task  ID: 
□Task  ID: 
□Task  ID: 
■Task  ID: 
□Task  ID: 
□Task  ID: 
□Task  ID: 
■Task  ID: 
■Task  ID: 
□Task  ID: 
□Task  ID: 
■Task  ID: 
■*Task  ID 
□Task  ID: 
■Task  ID: 
□Task  ID: 
□Task  ID: 
□  Task  ID: 


5827 

5832 

6366 

6286 

6427 

6488 

6534 

6923 

7580 

7695 

7778 

7915 

8123 

:  8966 

9141 

9424 

9552 

10264 

10289 
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Sample  Task  Description 


*Task  ID:  5827  Task  Name:  *Advanced  Thin  Film  Technology  for  Ultra-lightweight  Solar  Cell  Arrays 


GRC 


AST 


Point  of  Contact 


First  Name 

Last  Name 

E-Mail 

Phone 

Task  POC 

| 

Site  POC 

COTR 

Description 


*Objective  &  Description 


‘Technology  Disciplines  (Specific) 


1 - 

FY04 

Funding 

$K 

FY04 

FTE 

Power 

generation 

1 _ 1 

‘Technology  Application  Areas  &  Special  Categories 

e 

•  Terrestrial  Atmosphere/Surface  Systems 

•  Space  Technology/Spacecraft/Platform 

•  Space  Technology/Constellations 

•  Space  Technology/Planetary  Atmosphere/Surface  Systems 


Technology  Metrics 


Technology  Discipline:  Power 

Metric 

Technology  Metric 
Units 

Current  State  of  the  Art 
Value 

Planned 

Value 

No  metrics  entered  for  this  discipline 

Technology  Discipline:  Power  |  generation 


Metric 

Technology  Metric 
Units 

Current  State  of 
the  Art  Value 

Planned 

Value 

Efficiency 

% 

Mass  Specific 
Power 

w/kg 

Areal  Specific 
Power 

w/m2 

Specific  Cost 
($/w) 

Metric  Comments: 

Not  Specified 
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Milestones 


|trl 

Year 

Description 

i 

2004 

i 

2004 

2 

2004 

3 

2005 

6 

2012 

*Maturity  Time  Frame:  OFar 


Current  TRL  =  2 


FY _ 

*NASA  Profile  in  $K 
Civil  Service  FTE 


□ 

r 


[NASA  Distribution  Funding 

n 

Contracts  With  Whom  O 

ir 

□ 

Grants  With  Whom  Q 

r 

NASA  Field  Center  O 

r 

Other  (Space  Act,  etc.)  O 

r 

in 

partnership  Distribution  Funding 

1 

NASA  Contributions  Q 

r 

ir 

Other  Contributions  Q 

r 

r 

None 
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